👋 what kubernetes solution do folks use? anyone use k3s in production? at stripe, we’ve prototyped flyte with k3s but are now evaluating different solutions. curious to learn about your experiences!

Cc @Michael Lujan ?

I'm running it on k3s with metallb, traefik and longhorn... and don't see any reason to move to a different one.

👀 Bumping up @Hank Fanchiu's question ⬆️

k3s runs fine here so far, any specific questions @David Espejo (he/him)?

thanks for your input @Felix Ruess! Not much into the specifics, but more interested in having a sense of what other orgs are using to run Flyte/ML workloads? From what I can tell so far EKS/GKE/AKS (in that order) seem like common options, and k3s also makes a ton of sense

sure So if it's of interest: we run 3 master nodes in HA setup in VMs on our Proxmox cluster and the agent nodes (with various GPUs) are "bare-metal" machines. Kube-vip for control plane and metallb for other loadbalancer services. Longhorn for persistent storage (e.g. for Flyte postgres db), actual ML data in our MinIO cluster...

@Felix Ruess would you be open to sharing with the community your progress with Flyte and share thr journey during a community meeting

sure, but to be honest we can't run it in prod yet and run most things outside flyte still (e.g. because I can't assign affinity per task yet)

that's great @Felix Ruess! I'm sure we all can learn from your journey so far. @Tyler Su 👀

the affinity per task is coming soon