Hey all, my team is shifting some existing data pr...
# announcements
c
Hey all, my team is shifting some existing data processing code over to flyte and we are using map tasks to replace some existing parallel code, had a question about
functools.partial
and its (possible) flyte equivalent
. More details in thread
One pattern we're seeing emerge is we often have a task before our map task that prepares the list of inputs like:
Copy code
inputs = prepare_inputs(input_list=some_list, static_input_1 = 'value')
results = map_task(task_to_map)(i=inputs)
Where
prepare_inputs
basically just zips up the input_list with one or more static values. In non-flyte python, we use functools.partial for this. We could keep writing a helper task like
prepare_inputs
, but we were wondering if there was some more paradigmatic way of doing this in flyte?
.with_overrides
is clearly focused on execution parameters (e.g., its use with the spark task) so it doesn't seem like a good target for extending for this use case. It almost seems you like you could do something like functools.partial on the task itself in the workflow, and then add this to flytekit (would be interested in doing this if something like this doesn't already exist or is in the works, couldn't see anything in open issues)
👍 1
(Lmk if I should move this to onboarding?)
k
@Yee
y
@Calvin Leather do you mean if the underlying task has more than one input?
c
The underlying task has one static input that is the same for all executions, and then a list as the argument, for example
(and the static input comes from another task)
y
i think there was a decision made back when this was implemented to only support map tasks of one input
✔️ 1
i think we ran into some issue at implementation, and we did the more restrictive thing to start
but we should add it I think.
the partial concept extends to workflows kinda naturally because launch plans can have fixed inputs too
this is something we’ll need to do carefully i think but yeah, we’ve also talked about adding default inputs to tasks recently and i think this will be related.
c
Thanks for the background! I'll try to catch up on on the discussion of default inputs. This seems like a meaty feature for my team to contribute to immediately since we're still so new, but maybe in the coming weeks after we wet our feet with some other things
n
We should extend the docs here to include an example of using a task to prepare inputs to get ready for map task. Created an issue for this: https://github.com/flyteorg/flyte/issues/2382 @Smriti Satyan @Alekhya @Marc Paquette
👍 5
s
Sure @Niels Bantilan, we will work on it
170 Views