Would there be any way to set up gang scheduling within a project/domain currently? I've seen something similar for Kubeflow's training operators using https://github.com/volcano-sh/volcano and was wondering if it'd be possible to achieve for Flyte workflows.
For context, we'd like to create user-specific projects and enforce resource quotas on them in K8s; however, if we do this, we'd like to try to avoid multiple queued workflows contending for resources and let pods associated w/ a specific workflow get scheduled together. Another example of this would be https://github.com/palantir/k8s-spark-scheduler
07/15/2022, 7:06 PM
Yes you can change the scheduler to use
And you will have to use kube batch or yunikorn etc