Ailin Yu

    Ailin Yu

    1 week ago
    Hello! I am interested in using the published Grafana dashboards to monitor Flyte. But even though my Flyte instance has these pod annotations:
    <http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>
    <http://prometheus.io/port=15020|prometheus.io/port=15020>
    <http://prometheus.io/scrape=true|prometheus.io/scrape=true>
    And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    Check your Prometheus if it is successful
    Ailin Yu

    Ailin Yu

    1 week ago
    I see that the endpoint shows as UP and I don’t see any scrape errors logged. Is there another way you’d recommend I check as well? Edit: Also enabled debug logs, going to monitor those for a bit!
    Also from that response it sounds like the answer to my original question is probably “no” and it’s something I need to debug in prometheus! I just wanted to be sure of that, so I can take this offline then 👍
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    No Flyte propeller always generates metrics
    ya
    Ailin Yu

    Ailin Yu

    1 week ago
    Awesome! Thanks so much 🙏
    Dan Rammer (hamersaw)

    Dan Rammer (hamersaw)

    1 week ago
    @Ailin Yu did you get this fixed? Happy to dive in a little further if necessary.
    Ailin Yu

    Ailin Yu

    1 week ago
    Yes! Prometheus was configured to scrape the wrong endpoint! instead of using
    10254/metrics
    it was using the one for the istio sidecar. Thank you for following up 🙏