Hello everyone 🙂 I’m looking for some higher level advice on how to structure one of our workflows. We have a wf that runs a lot (10-100 million) of repetitive tasks, for which the execution logic is very stable. For now, we split the original input list into a more suitable number (10-100) of chunks and use

map_task

to map the chunks to equally many worker nodes. However, now I would like to add another task into the mix. The task would be a ShellTask, which would prefetch data into Flyte filesystem for the worker nodes to use in processing. Reasoning behind this is that currently the fetching of the data happens inside of the processing loop, which creates a sizeable I/O bottleneck. The problem is that according to the docs, one should not call another task from inside a mapped task. So I’m looking for a more flexible approach to distribute processing to multiple pods, which would allow calling tasks from inside the worker nodes. I’ve looked into

@dynamic

and subworkflows. Which would be better, or is there a better option? Thanks a ton

are your tasks sharing same data? or its each task needs different data

Each task has mostly unique data! Although there are some workfow-wise params given to the tasks as well

so is it like

map(a -> b)
where a and b are individual tasks

if they have mostly unique data, what is the advantage of having it in separate task, would it just punt the bottleneck to the other task

@Ketan (kumare3) thanks for the insights! I'll give dynamic a try

unless you share the storage space, it doesnt really work right? since each task is its own pod so will not have the fetched data