Thread
#announcements
    Calvin Leather

    Calvin Leather

    5 months ago
    Hey all, my team is shifting some existing data processing code over to flyte and we are using map tasks to replace some existing parallel code, had a question about
    functools.partial
    and its (possible) flyte equivalent
    . More details in thread
    One pattern we're seeing emerge is we often have a task before our map task that prepares the list of inputs like:
    inputs = prepare_inputs(input_list=some_list, static_input_1 = 'value')
    results = map_task(task_to_map)(i=inputs)
    Where
    prepare_inputs
    basically just zips up the input_list with one or more static values. In non-flyte python, we use functools.partial for this. We could keep writing a helper task like
    prepare_inputs
    , but we were wondering if there was some more paradigmatic way of doing this in flyte?
    .with_overrides
    is clearly focused on execution parameters (e.g., its use with the spark task) so it doesn't seem like a good target for extending for this use case. It almost seems you like you could do something like functools.partial on the task itself in the workflow, and then add this to flytekit (would be interested in doing this if something like this doesn't already exist or is in the works, couldn't see anything in open issues)
    (Lmk if I should move this to onboarding?)
    k

    katrina

    5 months ago
    @Yee
    Yee

    Yee

    5 months ago
    @Calvin Leather do you mean if the underlying task has more than one input?
    Calvin Leather

    Calvin Leather

    5 months ago
    The underlying task has one static input that is the same for all executions, and then a list as the argument, for example
    (and the static input comes from another task)
    Yee

    Yee

    5 months ago
    i think there was a decision made back when this was implemented to only support map tasks of one input
    i think we ran into some issue at implementation, and we did the more restrictive thing to start
    but we should add it I think.
    the partial concept extends to workflows kinda naturally because launch plans can have fixed inputs too
    this is something we’ll need to do carefully i think but yeah, we’ve also talked about adding default inputs to tasks recently and i think this will be related.
    Calvin Leather

    Calvin Leather

    5 months ago
    Thanks for the background! I'll try to catch up on on the discussion of default inputs. This seems like a meaty feature for my team to contribute to immediately since we're still so new, but maybe in the coming weeks after we wet our feet with some other things
    Niels Bantilan

    Niels Bantilan

    5 months ago
    We should extend the docs here to include an example of using a task to prepare inputs to get ready for map task. Created an issue for this: https://github.com/flyteorg/flyte/issues/2382 @Smriti Satyan @Alekhya @Marc Paquette
    Smriti Satyan

    Smriti Satyan

    5 months ago
    Sure @Niels Bantilan, we will work on it