Hello Flyte Community, I hope you're all doing well. I have some questions regarding subworkflows in Flyte, and I'd appreciate your guidance on the following topics: When to Use SubWorkflow: Could you please clarify the scenarios in which it's more beneficial to use SubWorkflow by importing a workflow versus using a reference workflow ( using reference_launch_plan / reference_workflow)? Are there any best practices or specific cases where one approach is preferred over the other ? any risks using reference_workflow on large scale as compared to imported workflow ? Handling SubWorkflow Failures: Is it possible to handle SubWorkflow failures from the parent workflow on node level ? we want to handle the SubWorkflow failure gracefully in the parent workflow and have its dependant tasks executed Thank you in advance for your help! cc @glamorous-rainbow-77959 @clever-exabyte-82294

hi! 1. reference workflow is useful when your workflow is present in a different project-domain; if you're able to import a workflow, then you can use it as a subworkflow directly 2. i believe so https://docs.flyte.org/en/latest/api/flytekit/generated/flytekit.WorkflowFailurePolicy.html

@tall-lock-23197

FAIL_AFTER_EXECUTABLE_NODES_COMPLETE

can allow all other nodes to complete, but how would we catch sub-workflow error inside the parent workflow? We need to act differently based on what happened there, and depending on which step has failed

if the subworkflow is in the same context as that of the current workflow, meaning if it isn't an external execution, then it should be possible.

Untitled

why would you want to run

task_g

when a node output that it depends on isn't available?

so for our use-case task_a, task_b could be inferencing model workflows for and task_g could be post processing on the generated recommendations, so even if task_a fails task_g can proceed, failed sub workflows can be grouped and re-triggered later.

okay, i think this has to work when `task_a`'s output is annotated with

Optional

in

task_g

. @high-accountant-32689 is this something we should support?

In case this is not supported, it’s also would be great to have an advice on if it’s possible to get around this in the current Flyte version?

So failures are failures. Failures will eventually fail the workflow unless you use map tasks which has a failure threshold. One option is to not make it a failure, return none? There is an on failure handler, but this will purposely not succeed the workflow today, Should we make it succeed? If so need help on understanding maybe like a rough prototype of a use case Last option is to use @eager - (experimental) but it allows arbitrary code and is an escape hatch

@freezing-airport-6809 we would be glad to jump on a call and discuss our usecase. Would Monday or Tuesday work? Which timezone are you in?

I am in pacific time. Let me dm You

@freezing-airport-6809 Trying to build a POC of Eager. However unable to create FlyteRemote needed for Eager Our setup is that we are using unauthenticated https internal Flyte endpoint on AWS, with no public access. Based on the document, I see that that we need client_secret_key. However we don't have this key as we dont need authentication. https://docs.flyte.org/en/latest/user_guide/advanced_composition/eager_workflows.html#remote-flyte-cluster-execution is there a way to Handle the case where the task is running in a Flyte cluster and needs to access the cluster itself?