Hello! I am interested in using the published Grafana dashboards to monitor Flyte. But even though my Flyte instance has these pod annotations:

<http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>

<http://prometheus.io/port=15020|prometheus.io/port=15020>

<http://prometheus.io/scrape=true|prometheus.io/scrape=true>

And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?

Check your Prometheus if it is successful

I see that the endpoint shows as UP and I don’t see any scrape errors logged. Is there another way you’d recommend I check as well? Edit: Also enabled debug logs, going to monitor those for a bit!

No Flyte propeller always generates metrics

Awesome! Thanks so much 🙏

@Ailin Yu did you get this fixed? Happy to dive in a little further if necessary.

Yes! Prometheus was configured to scrape the wrong endpoint! instead of using

10254/metrics

it was using the one for the istio sidecar. Thank you for following up 🙏