Hi all,
Are prometheus metrics configured by defau...
# ask-the-communityg
Guy Harel
02/13/2024, 2:09 PMHi all,
Are prometheus metrics configured by default when deploying using the flyte-binary chart?
It's my understanding that the metrics are exposed when the serviceMonitor is enabled for flyteadmin, however that configuration option is only found in the flyte-core chart and not flyte-binary.We deployed flyte using the flyte-binary chart, and prometheus using the prometheus-community/kube-prometheus-stack chart. I don't see that the flyte-binary chart has configured any ServiceMonitors..
d
David Espejo (he/him)
02/13/2024, 4:41 PMI see all the necessary pieces for monitoring available on
flyte-coreDavid Espejo (he/him)
02/13/2024, 4:42 PMYou'd have to add the extra metrics port using the
extraPortsDavid Espejo (he/him)
02/13/2024, 4:43 PMand then create an SM mimicking the config from the template in core
But this is me guessing, I haven't tried it
g
Guy Harel
02/13/2024, 4:45 PMThanks @David Espejo (he/him) , you referenced the service yaml, does the deployment need that port as well?
Guy Harel
02/13/2024, 4:47 PMBTW : Is there a downside to switching from flyte-binary to flyte-core?
d
David Espejo (he/him)
02/13/2024, 4:47 PMI see in flyte-core that the flyteadmin K8s service exposes, among other ports, the
http-metricsDavid Espejo (he/him)
02/13/2024, 4:49 PMIs there a downside to switching from flyte-binary to flyte-core?I think it depends on where's your Flyte instance deployed. In EKS for example, it would end up meaning a new IP backing the LB so a change in the DNS entry is needed. From the Flyte POV, the DB schemas are the same, maybe a backup is a good idea but the rest is stateless
g
Guy Harel
02/13/2024, 4:51 PMWe have the ip automated, so that's not an issue.. functionality wise is ir the same code, just running within different pods as opposed to squished together?
d
David Espejo (he/him)
02/13/2024, 4:52 PMyes, the SDK, API , everything is the same. It's just a different packaging
g
Guy Harel
02/13/2024, 4:53 PMGot it, thanks! 🙏
Guy Harel
02/13/2024, 4:54 PMI will try the extra ports and SM.
c
Cornelis Boon
04/09/2024, 9:38 AM@Guy Harel Did this approach work for you in the end? I'm looking into setting up monitoring via prom/grafana for flyte-binary right now. Or did you switch to flyte-core in the end?
g
Guy Harel
04/09/2024, 9:49 AMActually I don't get it to work yet but other things took priority..
c
Cornelis Boon
04/09/2024, 9:50 AMAh, too bad. Thank you for your answer. The approach does look promising
4 Views