Hi all,
Are prometheus metrics configured by defau...
# flyte-supportt
tall-ram-83532
02/13/2024, 2:09 PMHi all,
Are prometheus metrics configured by default when deploying using the flyte-binary chart?
It's my understanding that the metrics are exposed when the serviceMonitor is enabled for flyteadmin, however that configuration option is only found in the flyte-core chart and not flyte-binary.We deployed flyte using the flyte-binary chart, and prometheus using the prometheus-community/kube-prometheus-stack chart. I don't see that the flyte-binary chart has configured any ServiceMonitors..
👀 1
a
average-finland-92144
02/13/2024, 4:41 PMI see all the necessary pieces for monitoring available on
flyte-coreaverage-finland-92144
02/13/2024, 4:42 PMYou'd have to add the extra metrics port using the
extraPortsaverage-finland-92144
02/13/2024, 4:43 PMand then create an SM mimicking the config from the template in core
But this is me guessing, I haven't tried it
t
tall-ram-83532
02/13/2024, 4:45 PMThanks @average-finland-92144 , you referenced the service yaml, does the deployment need that port as well?
tall-ram-83532
02/13/2024, 4:47 PMBTW : Is there a downside to switching from flyte-binary to flyte-core?
a
average-finland-92144
02/13/2024, 4:47 PMI see in flyte-core that the flyteadmin K8s service exposes, among other ports, the
http-metricsaverage-finland-92144
02/13/2024, 4:49 PMIs there a downside to switching from flyte-binary to flyte-core?I think it depends on where's your Flyte instance deployed. In EKS for example, it would end up meaning a new IP backing the LB so a change in the DNS entry is needed. From the Flyte POV, the DB schemas are the same, maybe a backup is a good idea but the rest is stateless
t
tall-ram-83532
02/13/2024, 4:51 PMWe have the ip automated, so that's not an issue.. functionality wise is ir the same code, just running within different pods as opposed to squished together?
a
average-finland-92144
02/13/2024, 4:52 PMyes, the SDK, API , everything is the same. It's just a different packaging
t
tall-ram-83532
02/13/2024, 4:53 PMGot it, thanks! 🙏
tall-ram-83532
02/13/2024, 4:54 PMI will try the extra ports and SM.
🙌🏽 1
g
gentle-tomato-480
04/09/2024, 9:38 AM@tall-ram-83532 Did this approach work for you in the end? I'm looking into setting up monitoring via prom/grafana for flyte-binary right now. Or did you switch to flyte-core in the end?
t
tall-ram-83532
04/09/2024, 9:49 AMActually I don't get it to work yet but other things took priority..
g
gentle-tomato-480
04/09/2024, 9:50 AMAh, too bad. Thank you for your answer. The approach does look promising
7 Views