https://flyte.org
Join Slack
Hi all, Are prometheus metrics configured by default when deploying using the flyte-binary chart? It...
# flyte-support
t
Hi all, Are prometheus metrics configured by default when deploying using the flyte-binary chart? It's my understanding that the metrics are exposed when the serviceMonitor is enabled for flyteadmin, however that configuration option is only found in the flyte-core chart and not flyte-binary.We deployed flyte using the flyte-binary chart, and prometheus using the prometheus-community/kube-prometheus-stack chart. I don't see that the flyte-binary chart has configured any ServiceMonitors..
👀 1
a
I see all the necessary pieces for monitoring available on 
flyte-core
but not in binary
You'd have to add the extra metrics port using the 
extraPorts
in binary (see)
and then create an SM mimicking the config from the template in core But this is me guessing, I haven't tried it
t
Thanks @average-finland-92144 , you referenced the service yaml, does the deployment need that port as well?
BTW : Is there a downside to switching from flyte-binary to flyte-core?
a
I see in flyte-core that the flyteadmin K8s service exposes, among other ports, the 
http-metrics
- `10254`that then the SM is configured to scrape metrics from
Is there a downside to switching from flyte-binary to flyte-core?
I think it depends on where's your Flyte instance deployed. In EKS for example, it would end up meaning a new IP backing the LB so a change in the DNS entry is needed. From the Flyte POV, the DB schemas are the same, maybe a backup is a good idea but the rest is stateless
t
We have the ip automated, so that's not an issue.. functionality wise is ir the same code, just running within different pods as opposed to squished together?
a
yes, the SDK, API , everything is the same. It's just a different packaging
t
Got it, thanks! 🙏
I will try the extra ports and SM.
🙌🏽 1
g
@tall-ram-83532 Did this approach work for you in the end? I'm looking into setting up monitoring via prom/grafana for flyte-binary right now. Or did you switch to flyte-core in the end?
t
Actually I don't get it to work yet but other things took priority..
g
Ah, too bad. Thank you for your answer. The approach does look promising
9 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by