https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Ailin Yu

09/14/2022, 6:22 PM
Hello! I am interested in using the published Grafana dashboards to monitor Flyte. But even though my Flyte instance has these pod annotations:
<http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>
<http://prometheus.io/port=15020|prometheus.io/port=15020>
<http://prometheus.io/scrape=true|prometheus.io/scrape=true>
And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?
k

Ketan (kumare3)

09/14/2022, 8:07 PM
Check your Prometheus if it is successful
a

Ailin Yu

09/14/2022, 8:13 PM
I see that the endpoint shows as UP and I don’t see any scrape errors logged. Is there another way you’d recommend I check as well? Edit: Also enabled debug logs, going to monitor those for a bit!
Also from that response it sounds like the answer to my original question is probably “no” and it’s something I need to debug in prometheus! I just wanted to be sure of that, so I can take this offline then 👍
k

Ketan (kumare3)

09/14/2022, 9:31 PM
No Flyte propeller always generates metrics
ya
a

Ailin Yu

09/14/2022, 9:32 PM
Awesome! Thanks so much 🙏
d

Dan Rammer (hamersaw)

09/15/2022, 3:01 PM
@Ailin Yu did you get this fixed? Happy to dive in a little further if necessary.
a

Ailin Yu

09/15/2022, 5:16 PM
Yes! Prometheus was configured to scrape the wrong endpoint! instead of using
10254/metrics
it was using the one for the istio sidecar. Thank you for following up 🙏
99 Views