https://flyte.org
Join Slack
Hi everyone. Could anyone guide me on how to implement workflow finalizers in Flyte? For example, I ...
# flyte-support
g
Hi everyone. Could anyone guide me on how to implement workflow finalizers in Flyte? For example, I want to run some code to report status back to a slack channel that should run at the end of the workflow independently of any upstream exceptions that could have happened?
In Kubeflow, the similar concept is called ExitHandler
in Python, that would be 
try-finally
block
f
The workflow has a failure handler - but it’s invoked on failures only
If you only want to send notifications you can use launchplan notifications- there you can decide conditions
g
Notification mechanism is pretty cool, but unfortunately it’s limited to only Slack, Email and Pagerduty and does not allow customisation if I am not mistaken
Failure handler is a nice think that I did not see previously, thank you
Although what I need is finalizer and not notification. For example, to run compensation actions if anything needs rolling back due to failure
f
Ya failure handler is right then. You just have to repeat the same task twice once in failure handler and other times as the last task
Failure handler is only called on failure
g
@freezing-airport-6809 by any chance, does 
on_failure
support getting any context about which error did occur in upstream tasks?
f
Not today but this is planned - can you please write an issue and explain what type Of context would be useful. Especially if it is identifying what failed - is the exception useful or what
g
Of context would be useful. Especially if it is identifying what failed - is the exception useful or what
Any notion of upstream exception, trace to understand what error took place for reporting and monitoring purposes
@freezing-airport-6809 I fear that the 
on_failure
example is not working for me. I get something like this for my failure handler task whenever I deploy a workflow and try running it
Copy code
Collected Errors: 2
	Error 0: Code: TaskReferenceNotFound, Node Id: fn0, Description: Referenced Task [resource_type:TASK project:"flyte-poc" domain:"development" name:"universal_model_workflows.wf.send_failure" version:"feature-plat-17-eb45f920e89a77e87cf1a9eb38f30de640ce4e10" ] not found.
	Error 1: Code: WorkflowBuildError, Node Id: , Description: Collected Errors: 1
	Error 0: Code: TaskReferenceNotFound, Node Id: fn0, Description: Referenced Task [resource_type:TASK project:"flyte-poc" domain:"development" name:"universal_model_workflows.wf.send_failure" version:"feature-plat-17-eb45f920e89a77e87cf1a9eb38f30de640ce4e10" ] not found.
However, the task definition is there in my workflow file
I can’t make it work with a simple setup:
Copy code
@task(requests=Resources(cpu="100m", mem="512M"), retries=2)
def faulty_task():
    raise ValueError("I am flawed by nature")

@task
def on_error_task():
    print("I am the one who notices the fault.")

@workflow(on_failure=on_error_task)
def wf():
    faulty_task()
I am using Flyte v1.10.6
g
@glamorous-rainbow-77959 this feature is in v1.10.7
🆗 1
g
@glamorous-carpet-83516 I have just updated our cluster to 1.10.7, and the above example is still not working
I do not see any logs from 
on_error_task
It’s execution is not reflected anywhere, I do not see any additional pods being created
The execution looks like this to me:
f
Yes the Ui is missing the onfailure node
Cc @glamorous-carpet-83516
This will be fast followed
g
@glamorous-carpet-83516 @freezing-airport-6809 could you tell me if there’s any ETA for this update? Our pipeline code is a bit too complicated since we have to use try/except blocks in each task to mimic what 
on_error
should do CC @clever-exabyte-82294
👍 1
f
Cc @silly-toddler-37820 / @late-eye-50215 let’s get the finalize node in
👀 1
g
@freezing-airport-6809 @silly-toddler-37820 @late-eye-50215 hi everyone. Could you please confirm that this is fixed in 0.11? Also, is there any way to get exception details in the 
on_error
handler?
f
It is not. 1.11 has a major Ui update in beta. Once that is approved by the community the following release will have it
12 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by