Hello Flyte Community, I hope you're all doing wel...
# ask-the-community
n
Hello Flyte Community, I hope you're all doing well. I have some questions regarding subworkflows in Flyte, and I'd appreciate your guidance on the following topics: When to Use SubWorkflow: Could you please clarify the scenarios in which it's more beneficial to use SubWorkflow by importing a workflow versus using a reference workflow ( using reference_launch_plan / reference_workflow)? Are there any best practices or specific cases where one approach is preferred over the other ? any risks using reference_workflow on large scale as compared to imported workflow ? Handling SubWorkflow Failures: Is it possible to handle SubWorkflow failures from the parent workflow on node level ? we want to handle the SubWorkflow failure gracefully in the parent workflow and have its dependant tasks executed Thank you in advance for your help! cc @Kirill Dubovikov @divyank Agarwal
s
hi! 1. reference workflow is useful when your workflow is present in a different project-domain; if you're able to import a workflow, then you can use it as a subworkflow directly 2. i believe so https://docs.flyte.org/en/latest/api/flytekit/generated/flytekit.WorkflowFailurePolicy.html
k
@Samhita Alla
FAIL_AFTER_EXECUTABLE_NODES_COMPLETE
can allow all other nodes to complete, but how would we catch sub-workflow error inside the parent workflow? We need to act differently based on what happened there, and depending on which step has failed
s
if the subworkflow is in the same context as that of the current workflow, meaning if it isn't an external execution, then it should be possible.
n
Untitled
@Samhita Alla here is the example that we've tried where failure is being simulated for subworkflow_a, so it failed, now the task_g is dependant on subworkflow_a, in some cases user still might want to execute task_g without subworkflows_a's output and b, couldn't find a way handle the failure of subworkflow_a from main_workflow. Attached the screenshot and workflow code for reference failure handling from the main/parent workflow can provide flexibility and more control as one might not always have control over subworkflow's from different teams/modules while orchestrating large and complex workflows
s
why would you want to run
task_g
when a node output that it depends on isn't available?
n
so for our use-case task_a, task_b could be inferencing model workflows for and task_g could be post processing on the generated recommendations, so even if task_a fails task_g can proceed, failed sub workflows can be grouped and re-triggered later.
s
okay, i think this has to work when `task_a`'s output is annotated with
Optional
in
task_g
. @Eduardo Apolinario (eapolinario) is this something we should support?
k
In case this is not supported, it’s also would be great to have an advice on if it’s possible to get around this in the current Flyte version?
Hi @Samhita Alla @Eduardo Apolinario (eapolinario). Could someone guide us on error handling for SubWorkflows? Just re-iterating the question in case anyone has done something similar: We have a set of sub-workflows that can fail internally, but the parent workflow should gracefully handle errors and continue it’s execution. Parent workflow orchestrates execution of multiple inference pipelines with different logic of execution. Some of them are parallelizable, some are linear. And, as stated, any of them can fail. Parent workflow then needs to understand which ones failed, which completed successfully and act based on that
k
So failures are failures. Failures will eventually fail the workflow unless you use map tasks which has a failure threshold. One option is to not make it a failure, return none? There is an on failure handler, but this will purposely not succeed the workflow today, Should we make it succeed? If so need help on understanding maybe like a rough prototype of a use case Last option is to use @eager - (experimental) but it allows arbitrary code and is an escape hatch
Happy to jump on a call to understand in more detail
k
@Ketan (kumare3) we would be glad to jump on a call and discuss our usecase. Would Monday or Tuesday work? Which timezone are you in?
k
I am in pacific time. Let me dm You
d
@Ketan (kumare3) Trying to build a POC of Eager. However unable to create FlyteRemote needed for Eager Our setup is that we are using unauthenticated https internal Flyte endpoint on AWS, with no public access. Based on the document, I see that that we need client_secret_key. However we don't have this key as we dont need authentication. https://docs.flyte.org/en/latest/user_guide/advanced_composition/eager_workflows.html#remote-flyte-cluster-execution is there a way to Handle the case where the task is running in a Flyte cluster and needs to access the cluster itself?
Nevermind.. got it working.