https://flyte.org logo
#ask-the-community
Title
# ask-the-community
n

Nicholas Roberson

04/05/2023, 4:06 PM
Is there a recommended way to attach labels/annotations to pods/jobs in flyte such that they will show up in the prometheus metrics aggregations? Currently I am using the
labels
and
annotations
tags in a
LaunchPlan
however I am not seeing these show up in any of the metrics queries I am running against our prometheus server.
Additionally any recommended ways for custom pod names would also be appreciated.
y

Yee

04/05/2023, 6:15 PM
can you verify that the labels/annotations are showing up on the task pods?
n

Nicholas Roberson

04/05/2023, 6:58 PM
I have been able to get them in the task config however I am still waiting on getting access to
kubectl
so I can pull info from kubernetes api and confirm whether or not they have been propogated to the pods.
So this is what it looks like now after using the
@task(task_config=Pod(pod_spec=pod_spec, labels=labels, annotations=annotations, ...)
@Yee I do not see labels/annotations attached to logs in DataDog though. But again waiting on getting access to
kubectl
to see if these task labels are properly getting propogated.
y

Yee

04/05/2023, 7:19 PM
let’s confirm the labels/annotations on the pod first.
you can use pod templates also
n

Nicholas Roberson

04/05/2023, 7:20 PM
solid I will get back once I can do that.
y

Yee

04/05/2023, 7:20 PM
you don’t need to set it on the task
n

Nicholas Roberson

04/05/2023, 7:21 PM
Ah yes I have been doing that on LaunchPlan creation as well, since it seems like I wouldnt need to specify in as many places. However I havent used pod templates yet.
Okay so the labels are getting picked up
Copy code
kubectl -n flytetester-development get pod ************ -o json | jq .metadata.labels
{
  "app": "prometheus_exporter",
  "domain": "development",
  "interruptible": "false",
  "node-id": "n0",
  "owner": "nick",
  "project": "flytetester",
  ....
}
however queried from prometheus like
kube_pod_labels{pod="fe16583c0005d43eba06-n0-0"}
yields nothing
So this may be an issue with our prometheus not being set up to ingest certain metrics/labels
Just to close this out, issue was with our prometheus config, updated to scrape labels and all works now @Yee
y

Yee

04/07/2023, 9:25 PM
🙏
thank you
n

Nicholas Roberson

04/11/2023, 2:04 PM
Followup to this, is there any way to get a value pre-pended to the pod name in kubernetes? For example our pods now are named like
f227c8b9ffda446c2889-frpr4oka-0-dn8-0
which is the default, but could we prepend a value to the beginning of that name via some settings or whatnot.
3 Views