New RunLLM response generated: >>>*# Question* Hi community, regarding Prometheus metrics in flytepropeller, we've been emitting metrics in propeller at the task plugin level which contain metrics that have

_unlabeled

suffix added to it in addition to normal

success

and

failure

metrics such that there exist

success_unlabeled

for example. I can see that this is generated from

EmitUnlabeledMetric()

which is implemented in

promutils

of stdlib. What is the purpose of the split between labeled vs unlabeled? I can see that a metric is considered unlabeled if it doesn't exist within the

taskMetricsMap

, but it isn't too clear to me why a metric name would be in the map in the first place or not. # Answer The split between labeled and unlabeled metrics in Flyte is primarily for performance and resource management. According to this discussion, Prometheus best practices recommend keeping metric labels to very low cardinality to avoid excessive memory utilization. Metrics with high cardinality can lead to significant overhead and resource consumption, which can be problematic in large-scale systems. The

EmitUnlabeledMetric()

function in

promutils

is used to emit metrics without additional labels, ensuring that overall system performance metrics are collected without the overhead of high cardinality labels. This is particularly useful for capturing general performance metrics while keeping the system efficient. Would you like more detail on how metrics are managed in Flyte, or a specific example of how to use

EmitUnlabeledMetric()

? Sources: - Prometheus Metric Labels Discussion - Flyte Metric Option Implementation Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1718660253.160029 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.