Is it possible to request resources for a task dynamically? We have a task that depending on the input could require a different amount of resources.
h
hallowed-mouse-14616
12/02/2022, 6:23 PM
Yes, there is a
with_overrides
that allows you to set task resources. To do this dynamically you'll have to run the task within a dyanmic task, which can then analyze the inputs and assign correct resources. For example:
Very toy example, probably syntactically incorrect, but I'm sure you get the idea.
hallowed-mouse-14616
12/02/2022, 6:24 PM
This, or course, then requires two Pods to start. One for the dynamic task to dynamically compile the new workflow closure, and then a second to actually execute the task. So there is a minimal amount of overhead.