Hi all is there a canonical way for dynamic workflows to col

Question

Hi all is there a canonical way for dynamic workflows to collect only succesful sub tasks I have a task A that may or may not succeed I have a dynamic workflow D that launches a bunch of instances of A with various parameters I want to collect only the successful results of A aggregate them and then return that result from D I ve seen discussions of an `allowed failure ratio` parameter but that seems to have been removed when moving away from sdk tasks There s the `WorkflowFailurePolicy` but while that would allow every sub task to run it doesn t seem that it would allow D to run My current approach is to wrap A in a `try except` that returns an empty result but D can t inspect the results to ignore A so the aggregator has to decide if this is a valid result or not Does anyone have a better idea for how to return what is effectively a I failed but it s okay just ignore me signal

Yee · Answer

no you re going to need an aggregator some how

Yee · Answer

if you use an optional type that might make the logic cleaner

Yee · Answer

and if all the subtasks you can use a map task which still has an allowed failure ratio

Yee · Answer

if you specify a failure ratio lt 1 the output of the map task by default becomes a list of optionals

Yee · Answer

but you are still on the hook to do the proverbial filter by is not none

Sasha Ayvazov · Answer

Cool that makes sense Thanks Just for my understanding the reason it s like this is that the dynamic task is run at instantiation and then returns the DAG and the actual promise objects don t get collected until that goes into another task

Yee · Answer

that is what happens yeah but the reason that propeller only knows how to pass inputs and outputs it doesn t know how to pass if not None or other things that people have asked for in the past like add 1 to an output before passing it on for small use cases you can use conditional workflows