https://flyte.org
Join Slack
~Is there a recommended way to attach labels/annotations to pods/jobs in flyte such that they will s...
# flyte-support
q
Is there a recommended way to attach labels/annotations to pods/jobs in flyte such that they will show up in the prometheus metrics aggregations? Currently I am using the 
labels
and 
annotations
tags in a 
LaunchPlan
however I am not seeing these show up in any of the metrics queries I am running against our prometheus server. Additionally any recommended ways for custom pod names would also be appreciated.
t
can you verify that the labels/annotations are showing up on the task pods?
q
I have been able to get them in the task config however I am still waiting on getting access to 
kubectl
so I can pull info from kubernetes api and confirm whether or not they have been propogated to the pods.
So this is what it looks like now after using the 
@task(task_config=Pod(pod_spec=pod_spec, labels=labels, annotations=annotations, ...)
@thankful-minister-83577 I do not see labels/annotations attached to logs in DataDog though. But again waiting on getting access to 
kubectl
to see if these task labels are properly getting propogated.
t
let’s confirm the labels/annotations on the pod first.
you can use pod templates also
q
solid I will get back once I can do that.
t
you don’t need to set it on the task
q
Ah yes I have been doing that on LaunchPlan creation as well, since it seems like I wouldnt need to specify in as many places. However I havent used pod templates yet.
Okay so the labels are getting picked up
Copy code
kubectl -n flytetester-development get pod ************ -o json | jq .metadata.labels
{
  "app": "prometheus_exporter",
  "domain": "development",
  "interruptible": "false",
  "node-id": "n0",
  "owner": "nick",
  "project": "flytetester",
  ....
}
however queried from prometheus like 
kube_pod_labels{pod="fe16583c0005d43eba06-n0-0"}
yields nothing
So this may be an issue with our prometheus not being set up to ingest certain metrics/labels
Just to close this out, issue was with our prometheus config, updated to scrape labels and all works now @thankful-minister-83577
t
🙏
thank you
q
Followup to this, is there any way to get a value pre-pended to the pod name in kubernetes? For example our pods now are named like 
f227c8b9ffda446c2889-frpr4oka-0-dn8-0
which is the default, but could we prepend a value to the beginning of that name via some settings or whatnot.
174 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by