https://flyte.org
Join Slack
Hey I'm trying to setup prometheus/grafana for my `flyte-binary` deployment on GKE. As mentioned <he...
# flyte-deployment
g
Hey I'm trying to setup prometheus/grafana for my 
flyte-binary
deployment on GKE. As mentioned here, it's not providing a port + metrics/a servicemonitor out of the box like with 
flyte-core
, so as per instruction I've added a port and a 
podMonitoring
(using GKE's managed prom service) . Don't seem to be getting any metrics yet. When looking at 
flyte-core
I see that there are two service monitors configured, 1 for admin and one for propeller. Wondering if adding just 1 port would be enough to handle both in this case and if the ports/paths are different
a
Wondering if adding just 1 port would be enough to handle both in this case and if the ports/paths are different
Agree, I don't think that's enough. The SM would be very beneficial as they are for flyte-core. I'm not even sure if 
flyte-binary
is completely instrumented to surface all metrics. Could you file an issue for this?
g
https://github.com/flyteorg/flyte/issues/3758 Seems like at least one person has gotten some metrics working on flyte-binary (but not the propeller metrics_
a
Right, I guess can iterate from there
g
Still, would be nice to have a guide/set of instructions on what is necessary to get the already available metrics working
a
That's exactly part of what I'm working on now, for flyte-core at least. I'll add it to the docs real soon
g
👍 Awesome
a
@gentle-tomato-480 that'd be a great contribution 🙇🏽
question for early feedback: the process I followed to consume the Grafana dashboards for propeller makes use of the 
prometheus-operator
, are you also using it in your cluster?
g
No, I'm currently trying to use https://cloud.google.com/stackdriver/docs/managed-prometheus (because it was enabled on the cluster). Just curious to see if it's easy to setup. Seemingly, should be able to set it up with Grafana too: https://cloud.google.com/stackdriver/docs/managed-prometheus/query#ui-grafana
Though, long term, it's probably better/easier for me to switch to just prometheus-operator/install prometheus in the cluster instead of having it managed
@average-finland-92144 I think I more or less know which configs are necessary for metrics that are currently missing in flyte-binary. I'm less familiar with how flyte (and k8s in general probably) uses configmaps. In addition to the port I added, I essentially need to add somewhere:
Copy code
(in a core.yaml)
propeller:
   metrics-prefix: flyte
   prof-port: 10254
(in a server.yaml)
adminServer:
   flyteadmin:
      metricsScope: flyte
      profilerPort: 10254
(in a server.yaml)
scheduler:
    metricsScope: flyte
    profilerPort: 10254
Could I just add these under 
configuration.inline
in the values file? Does it matter that these are not in some section called core.yaml or server.yaml respectively?
It seems like it, as what is defined under 
cluster_resources.yaml
in the generated example for gcp on github is defined in 
inline-config.yaml
in my cluster 🤔
a
@gentle-tomato-480
Does it matter that these are not in some section called core.yaml or server.yaml respectively?
I don't think so. Those are templates that seem to feed the configmaps used by flyteadmin and flytpropeller. Sometimes that complicates things when you need to insert keys from other sections to those templates, but that's not the case with flyte-binary. As long as you keep the sections consistent with the ones defined here, for example, for flyteadmin, you should be able to fold everything inside 
inline
g
@average-finland-92144 Great. Btw, should I be able to see metrics in the logs as well or only in the prometheus/grafana?
a
I don't think they're available in logs, but maybe increasing the verbosity level? btw, I added some notes to the monitoring section. Your feedback is appreciated https://docs.flyte.org/en/latest/deployment/configuration/monitoring.html#use-published-dashboards-to-monitor-flyte-deployment
g
Thanks. Will have a look
27 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by