:wave: what kubernetes solution do folks use? anyo...
# flyte-support
m
👋 what kubernetes solution do folks use? anyone use k3s in production? at stripe, we’ve prototyped flyte with k3s but are now evaluating different solutions. curious to learn about your experiences!
f
Cc @tall-continent-50414 ?
q
I'm running it on k3s with metallb, traefik and longhorn... and don't see any reason to move to a different one.
👀 1
🔥 1
a
👀 Bumping up @miniature-postman-38835's question ⬆️
👍 1
q
k3s runs fine here so far, any specific questions @average-finland-92144?
a
thanks for your input @quaint-diamond-37493! Not much into the specifics, but more interested in having a sense of what other orgs are using to run Flyte/ML workloads? From what I can tell so far EKS/GKE/AKS (in that order) seem like common options, and k3s also makes a ton of sense
q
sure So if it's of interest: we run 3 master nodes in HA setup in VMs on our Proxmox cluster and the agent nodes (with various GPUs) are "bare-metal" machines. Kube-vip for control plane and metallb for other loadbalancer services. Longhorn for persistent storage (e.g. for Flyte postgres db), actual ML data in our MinIO cluster...
🤩 1
and Grafana agent ships metrics and logs to Prometheus and Loki, viewing Flyte task logs via Grafana/Loki works nicely
🔥 2
f
@quaint-diamond-37493 would you be open to sharing with the community your progress with Flyte and share thr journey during a community meeting
q
sure, but to be honest we can't run it in prod yet and run most things outside flyte still (e.g. because I can't assign affinity per task yet)
And it seems that we are a bit atypical Flyte users, with only a few container tasks in a workflow so far. Not using most of the "fancy" Flyte features... But sure, I would be up to talking about it...
🔥 1
a
that's great @quaint-diamond-37493! I'm sure we all can learn from your journey so far. @wonderful-afternoon-77766 👀
👍 1
f
the affinity per task is coming soon
160 Views