Hey team! Wondering if there is a paved-path towards task-level notifications in Slack/email? Looking to do things like: • Notify when task status changes • Notify when task has been in a particular state (i.e. Queued) for a long time Flyte has notifications capabilities - but it seems these are limited to the execution phase at the workflow level. I see that work is ongoing to better support task-level notifications, but curious to know if teams have other solutions in the meanwhile?

Hey 🙂 Do you have prometheus and grafana in your cluaster? Flyte exposes metrics to prometheus, which include task state, and it is possible to set up notifications based on the metrics it exposes from your cluster to Slack. https://akhil47blog.wordpress.com/2021/03/11/configuring-alerts-in-prometheus-and-grafana-for-slack-and-email/

Thanks Lev! I'll look into this.

1.

flyte-propeller-srv

2. Can base it on difference between vs a failed workflow from a few seconds/minutes before using an offset

flyte:propeller:all:workflow:failure_duration_ms_count{project="default", domain="staging"})

Nice! Would this only be for

failed

workflow status, or are there metrics in

flyte-propeller-srv

that correspond to an individual task statuses (i.e. Queued, Running, etc.)? Apologies if I'm using these terms wrong, I'm a bit new to Flyte so learning the ropes!

Just checked and there are some under

flyte-admin-srv

also that might be relevant, my bad. check out prometheus and play around with the queries see what you can find that suits your needs 🙂

Thanks so much Lev, will give these a try and let you know

GL & HF 💪