Hey all, I want to emit model performance metrics like accuracy, precision, and recall to Prometheus to track model performance over time, we have Prometheus setup for basic flyte metrics but emitting custom stats at task runtime seems like a heavy effort involving using ExecutionParameters statsd client, setting up Prometheus statsd exporter, customizing the statsdconfig, is there any way to return metrics to propeller and have them emitted from flyte backend instead of using statsd from task pods

Yes use Flyte decks

Thanks Ketan, we are heavily using flyte decks and Jupyter notebooks, but would like these metrics on a grafana dashboard to show how a model generated in a flyte workflow performs over time (predeploy performance testing), say new training data comes in the dashboard and grafana alerts can show us if there is data drift/concept drift based on some threshold in the metrics If there’s anyone who has setup the statsd client in current_context() and Prometheus statsd exporter in the internal channel id definitely be interested in joining and learning more

statsd with prometheus is weird honestly as prom uses a pull based model

Yes, yeah mlflow seems better tailored to ml metrics than Prometheus/grafana, thanks for the ideas and support

@brief-egg-3683 also you can simpy use mlflow decorator and do not need to install mlflow