https://flyte.org logo
#ask-the-community
Title
# ask-the-community
j

Jakub Urban

03/05/2024, 10:19 PM
Hello! We’re currently trialing Flyte for some pipelines and one feature we’d make use of are dynamic workflows / map tasks with (run-time) dynamic resource requests. I was thinking of a combination of map_task and with_overrides, something like
Copy code
results = map_task(task)(data=input_data_iterable).with_overrides(limits=limits_iterable)
where
limits_iterable
would be an iterable of resource limits per (mapped) task. I haven’t been able to find out whether this is possible, could only find a discussion here where imho
with_overrides
applies the same overrides for all mapped tasks. I’m actually wondering whether something like this could work:
Copy code
@dynamic
def workflow():
    input_data_iterable = get_inputs()
    results = []
    for input_data in input_data_iterable:
        results.append(task(input_data).with_overrides(**get_overrides(input_data)))
Any help would be appreciated, thanks!
s

Samhita Alla

03/06/2024, 1:26 PM
what's
input_data
?
could only find a discussion here where imho
with_overrides
applies the same overrides for all mapped tasks.
that's correct.
j

Jakub Urban

03/06/2024, 2:20 PM
input_data
would just be a generic
task
parameter.
Thanks for getting back to me - will assume my observation correct though try to dig a bit deeper hopefully 🙂
s

Samhita Alla

03/06/2024, 4:58 PM
okay, i think your approach should work! let me know if you encounter any issues.