We are deploying flyte using the flyte binary helm chart

Question

We are deploying flyte using the flyte binary helm chart document says that Flyte Backend is written in Golang and exposes stats using Prometheus We don t see prometheus annotations `` `` `` being applied automatically Also we can find any config setting to enable this Should this be happening automatically on deploy or we need to change some config setting Also found where port number is mentioned but the flyte binary pod only exposes 8088 8089 and 9443 not 10254 should that be exposed somehow as well

jeev · Answer

yes 10254 looks like it should be exposed < Nandakumar Raghu>

jeev · Answer

as for the annotations that s a bit of an implementation detail based on how your prometheus scrape config is set up you can add it if your prometheus is configured to discover workloads based on those annotations another way to do it if you are using the prometheus operator is with pod service monitors <https github com prometheus operator prometheus operator blob main example user guides getting started example app pod monitor yaml>

Nandakumar Raghu · Answer

Any idea why port 10254 is not exposed from the pod Is there a config or setting somewhere we can check

jeev · Answer

you mean as a `containerPort`

Nandakumar Raghu · Answer

yes

Nandakumar Raghu · Answer

If a port is exposed I should be able to see it when I describe the pod correct

jeev · Answer

it needs to be explicitly added <https github com flyteorg flyte blob 14171fd84d171a609e619a214fa36695fc79990b charts flyte binary templates deployment yaml L179|here>

Nandakumar Raghu · Answer

Ok thanks I ll try that out

Nandakumar Raghu · Answer

Hi < jeev> I was able to expose the metrics port and see metrics in datadog the datadog agent is set to scrape metrics from pods that are annotated with prometheus annotations I am trying to set up the core metrics dashboard shown However I am not able to find the metrics referenced in the grafana json on datadog The metrics referenced in the json are of the format ```flyte propeller all free workers count flyte propeller all round abort error 5m flyte propeller all round system error unlabeled 5m flyte propeller all node plugin failure unlabeled flyte propeller all node plugin success unlabeled flyte propeller all round raw unlabeled ms 5m flyte propeller all round raw ms 5m flyte propeller all round panic unlabeled 5m flyte propeller all collector flyteworkflow flyte propeller all metastore cache hit flyte propeller all metastore cache miss flyte propeller all metastore head failure unlabeled``` But on datadog when I search metrics for Propeller I can only see these ```flyte admin admin builder flytepropeller build failures count flyte admin admin builder flytepropeller build successes count flyte admin admin execution manager propeller failures count``` 1 Why aren t we seeing all the metrics for propeller for metastore plugin etc and 2 Why are the metrics names different is there some aggregation happening somewhere

jeev · Answer

hmm it s possible that there is a bug in flyte binary that isn t properly exposing all the metrics this isn t an issue in flyte core since the services are separate

jeev · Answer

can you open an issue < Nandakumar Raghu>

Ketan (kumare3) · Answer

< jeev> I think you are right maybe we have to register multiple handles

jeev · Answer

or merge

Nandakumar Raghu · Answer

Opened an issue <https github com flyteorg flyte issues 3758>

Brian Tang · Answer

< Nandakumar Raghu> did you add it under

Brian Tang · Answer

gt it needs to be explicitly added < jeev> we installed flyte binary using helm in eks and utilising flyreorg s repo if this needs to be explicitly added does that mean we ll need to manage the chart on our own since `ports` isn t exposed via `values yaml`

jeev · Answer

< Brian Tang> ideally we can just fix the chart but we haven t gotten to this unfortunately

Brian Tang · Answer

< jeev> i can create a PR if that s something that would be desired something like this ``` deployment yaml ports name http containerPort 8088 name grpc containerPort 8089 name webhook containerPort 9443 if Values deployment profilerPort name profiler containerPort Values deployment profilerPort end values yaml deployment profilerPort 10254``` or just remove the `if` and hardcode it like the other ports in `deployment yaml`

jeev · Answer

hardcoding is probably fine the issue though is that not all the metrics are currently available through a single port