https://flyte.org logo
#ask-the-community
Title
# ask-the-community
s

Sherwyn Dsouza

03/21/2024, 4:42 PM
Hey team! Wondering if there is a paved-path towards task-level notifications in Slack/email? Looking to do things like: ā€¢ Notify when task status changes ā€¢ Notify when task has been in a particular state (i.e. Queued) for a long time Flyte has notifications capabilities - but it seems these are limited to the execution phase at the workflow level. I see that work is ongoing to better support task-level notifications, but curious to know if teams have other solutions in the meanwhile?
l

Lev Udaltsov

03/21/2024, 8:54 PM
Hey šŸ™‚ Do you have prometheus and grafana in your cluaster? Flyte exposes metrics to prometheus, which include task state, and it is possible to set up notifications based on the metrics it exposes from your cluster to Slack. https://akhil47blog.wordpress.com/2021/03/11/configuring-alerts-in-prometheus-and-grafana-for-slack-and-email/
s

Sherwyn Dsouza

03/21/2024, 9:14 PM
Thanks Lev! I'll look into this.
Lev, this route looks super promising! Just wanted to follow up: 1. Do you know whether these metrics (task state) are exposed via
flyte-propeller-srv
or
flyte-admin-srv
or something else? 2. Do you know what the specific name of the metric I might be looking for is?
l

Lev Udaltsov

03/22/2024, 4:04 PM
1.
flyte-propeller-srv
2. Can base it on difference between vs a failed workflow from a few seconds/minutes before using an offset
flyte:propeller:all:workflow:failure_duration_ms_count{project="default", domain="staging"})
s

Sherwyn Dsouza

03/22/2024, 4:06 PM
Nice! Would this only be for
failed
workflow status, or are there metrics in
flyte-propeller-srv
that correspond to an individual task statuses (i.e. Queued, Running, etc.)? Apologies if I'm using these terms wrong, I'm a bit new to Flyte so learning the ropes!
l

Lev Udaltsov

03/22/2024, 9:22 PM
Just checked and there are some under
flyte-admin-srv
also that might be relevant, my bad. check out prometheus and play around with the queries see what you can find that suits your needs šŸ™‚
s

Sherwyn Dsouza

03/25/2024, 3:27 PM
Thanks so much Lev, will give these a try and let you know
l

Lev Udaltsov

03/25/2024, 3:28 PM
GL & HF šŸ’Ŗ
3 Views