Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello! I am interested in using the <https://docs.flyte.org/en/latest/deployment/cluster_config/monitoring.html#use-published-dashboards-to-monitor-flyte-deployment|published Grafana dashboards> to monitor Flyte. But even though my Flyte instance has these pod annotations: `<http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>`
`<http://prometheus.io/port=15020|prometheus.io/port=15020>`
`<http://prometheus.io/scrape=true|prometheus.io/scrape=true>`
And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?

Check your Prometheus if it is successful 

I see that the endpoint shows as UP and I don’t see any scrape errors logged. Is there another way you’d recommend I check as well?
Edit: Also enabled debug logs, going to monitor those for a bit!

Also from that response it sounds like the answer to my original question is probably “no” and it’s something I need to debug in prometheus! I just wanted to be sure of that, so I can take this offline then :thumbsup:

No Flyte propeller always generates metrics

<@U03PG15FHV2> did you get this fixed? Happy to dive in a little further if necessary.

Yes! Prometheus was configured to scrape the wrong endpoint! instead of using `10254/metrics` it was using the one for the istio sidecar. Thank you for following up :pray: