Hi community, regarding Prometheus metrics in flytepropeller, we've been emitting metrics in propeller at the task plugin level which contain metrics that have

_unlabeled

suffix added to it (

flyte:propeller:plugin:databricks_success_unlabeled

for example). I can see that this is generated from

EmitUnlabeledMetric()

which is implemented in

promutils

of stdlib. What is the purpose of the split between labeled vs unlabeled? I can see that a metric is considered unlabeled if it doesn't exist within the

taskMetricsMap

, but it isn't too clear to me why a metric name would be in the map in the first place or not.

did you get an answer to this btw?

No I did not

cc @high-park-82026

Another request for follow up on this

I'm not too familiar with the distinction between the two. My guess is this is done to limit the size of the metrics to improve performance. Smaller metric = less storage + processing overhead

I'm speculating, but I assume labeled might be referring to Prometheus labels to allow for multidimensional metrics, and unlabeled would mean that it's a single dimensional metric.

I looked at this a few weeks ago btw… i came to the same conclusion that you did, which is that it doesn’t necessarily make any sense why that’s there.

Yea, what is bothersome is now, on our metric emissions for flytepropeller, we have both labeled and unlabeled, where unlabeled emits just a bit more frequently, but they have nearly the same values as each other. I'm inclined to just consolidate the two on our end.

I'm inclined to just consolidate the two on our end.

I guess that's friction that the flyte code shouldn't introduce. Would you mind filing an Issue to better track/discuss this?

Sure, I'll open an issue