Hello! We’re currently trialing Flyte for some pipelines and one feature we’d make use of are dynamic workflows / map tasks with (run-time) dynamic resource requests. I was thinking of a combination of map_task and with_overrides, something like

results = map_task(task)(data=input_data_iterable).with_overrides(limits=limits_iterable)

where

limits_iterable

would be an iterable of resource limits per (mapped) task. I haven’t been able to find out whether this is possible, could only find a discussion here where imho

with_overrides

applies the same overrides for all mapped tasks. I’m actually wondering whether something like this could work:

@dynamic
def workflow():
    input_data_iterable = get_inputs()
    results = []
    for input_data in input_data_iterable:
        results.append(task(input_data).with_overrides(**get_overrides(input_data)))

Any help would be appreciated, thanks!

what's

input_data

?

input_data

would just be a generic

task

parameter.

okay, i think your approach should work! let me know if you encounter any issues.