Hello,
I have a workflow that will request different memory for task depending on the workflow function’s parameter value.
e.g. I know below will not work. I just want to illustrate my use case.
Copy code
@workflow
def wf(region: str = 'us') -> None:
mem = '80Gi' if region == 'us' else '40Gi'
train(region=region).with_overrides(requests=Resources(mem=mem)) # train() is a task
How can I request dynamic resource values and re-use the same workflow function, so that I don’t have to create separate wf_us() & wf_other_regions(), etc. since I have a lot of tasks in the workflow that will be cut and pasted. (edited)
f
freezing-boots-56761
06/13/2023, 2:29 AM
You can use a dynamic task for this to materialize the final DAG with overrides for now. There is also an RFC for config overrides coming.
r
rich-garden-69988
06/13/2023, 1:51 PM
If it only has a few branches, you could also consider using Flytekit conditionals right?
s
salmon-refrigerator-32115
06/13/2023, 5:59 PM
@rich-garden-69988, conditional works for me. Thanks!