how best can one set the GPU quota for a project/domain in flyte? 🧵

hey @rapid-artist-48509 you could use templates for the ClusterResourceManager which accepts spec of K8s objects, like a ResourceQuota https://github.com/flyteorg/flyte/blob/f2a1ad7d49a8b0429d4b51e42c56edd0a8bb5666/charts/flyte/values.yaml#L655-L666 I haven't tried it but AFAICT the keys you use here (like

projectQuotaCpu

) are arbitrary so you could define

projectQuotaGPU

Thanks! hmm those are k8s quotas, which I don't need (I don't have e.g. autoscalers). I wonder if Flyte / Flyte Propeller will observe those limits tho and queue accordingly? I tried via the

flytectl update cluster-resource-attribute --attrFile cra.yaml

way and: • I couldn't see how to spec GPUS ... maybe it's just the nvidia gpu tag? • I believe I was using k8s units as expected but yeah things didn't work

Hey Paul So I don't think propeller enforces resourceQuotas, otherwise it would become a scheduler layer on top the K8s scheduling logic. As mentioned you can request the K8s api server to create a resourceQuota through Flyte but Pod placement decisions in response of resource availability is K8s responsibility. Having said that, you could create a resourceQuota that includes `hard.requests.nvidia.com/gpu: "<int`>"

Hey @average-finland-92144 thank you so much! I'm learning a lot here!! Yeah it makes sense that the propeller doesn't somehow double the k8s quota logic. That said I think I may have lead this inquiry astray a bit ... My end goal is for users to run lots of workflows

pyflyte run

, so many that all the resources get used and workflows have to queue up. Right now, the Flyte WebUI always shows these queued executions as "Running" even tho they are effectively queued by K8s (and pods are in the "pending" state). This behavior is confusing to users, as they think all their jobs have started and yet clearly they are stuck in line. So my questions I guess are: • What can I do, if anything, to get the workflows to properly display as "queued" in the executions view? I guess I thought that specifying quotas would help, but is there any other way? I think Ketan said that once a workflow / task gets sent to k8s, it's always "Running" according to Flyte. So I was trying to see how to keep workflows from getting dispatched w/out respect to resources. • If there really isn't any facility for the above bullet point, would perhaps my only avenue to be e.g. a feature request where perhaps workflows can report the actual pod state, i.e. "Pending" ? At the end of the day, a use case is like "the user kicked off 100 workflow runs at night, in the morning 90 were done or running and user wants to decide if the other 10 should just be terminated. But it's unclear if the other 10 ever started"

what do you see in the

Timeline

tab of the UI for one of those "Running" tasks? When you hover over the execution bar it displays phases. Also, this dashboard includes a metric for Tasks whose Pod is in

Pending

state in K8s: https://grafana.com/grafana/dashboards/22146-flyte-user-dashboard-via-prometheus/

Let me debug one more time, but I believe the timeline always just says "running" with wall clock time of when the task was sent to k8s. E.g. if I have an image pull that takes 2-3 minutes, that also just shows up in the timeline as if the task itself was running for 2-3 minutes. This is like a hello world task. (Also FWIW

TASK_RUNTIME

in the Timeline view is consistently showing like "-1hr" for a task that took about 1-2 sec to run, and perhaps 1-3 minutes full wallclock time (e.g. slow image pull). I like the idea of Grafana for the cluster, but that was really hoping the Flyte WebUI could just show if some execution is "later in the queue". Thanks for "fail fast" link, but yeah we definitely want to have a long queue of stuff to compute, like a CI system. Thanks again for all the pointers!! I guess none of this would change if we switch from the single-binary / flyte-binary helm chart to the multi-chart deployment?

you can get it using flyte apis

Thanks @freezing-airport-6809! yeah etcd is a gift and a curse ... thank you, i'll look at Cloud Events! To be clear: my immediate situation tho is users firing off lots of single-node workflows (rather than a few workflows with many map_tasks). And those workflows never show up as "queued" even tho k8s keeps them stuck in Pending. And I had expected the propeller would not actually dequeue them so fast, i.e. not dequeue more 1-GPU jobs than the cluster has GPUs (and I think I have my cluster resources configured correctly in my flyte helm chart)

You can do that by limiting the quota

I have tried that! And tried to document my steps in this thread, tho I apologize if it's not clear. The symbol

projectQuotaGPU

(read: GPU quota) in particular is not in Flyte OSS, so I wonder if it's just Union-only.