Hello Flyte folks, its me again 😆 Is there anyway we can abort the workflow fast if there is a problem with dynamic workflow compilation in flytepropeller. So right now, if there is issue such as mismatching input/output type or some unbounded variables it shows FAILED in console but we still have to wait till flytepropeller exhausts the system retry. How can i do early exit in flytepropeller Also if the dynamic workflow cannot be compiled it should fail and not retry until exhaustion right?. Here is one example

Workflow[balrog:development:balrog_pipeline.workflows.metrics.metrics_workflow.metrics_workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: Failed to check Catalog for previous results: unexpected artifactData: [o0] type: [collection_type:<union_type:<variants:<map_value_type:<blob:<> > > variants:<map_value_type:<union_type:<variants:<blob:<> structure:<tag:"FlyteFilePath" > > > > > > > ] does not match any task output type: [collection_type:<map_value_type:<union_type:<variants:<blob:<> structure:<tag:"FlyteFilePath" > > variants:<simple:NONE structure:<tag:"none" > > > > > ]

Hey @Jay Ganbat, do you want to configure system retries?

hi @Samhita Alla yeah for sure, do you have any doc on how to do that?

https://docs.flyte.org/en/latest/deployment/configuration/generated/flytepropeller_config.html#maxretries-int: you can configure this value.

hmm is it different than the

max-workflow-retries

also i was wondering if there is a way to stop right away since there is really no point in keep retrying this type of validation errors right

cc @Dan Rammer (hamersaw)

@Jay Ganbat as you suggest, we should probably update this error type to a non recoverable error so that it doesn't retry. Also, I suspect this error will be solved by the union fix from earlier.

thank you very much, yeah this error also happens in some of the malformed dynamic task, i just picked the above error because it was open in a tab haha