Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Would there be any way to set up gang scheduling within a project/domain currently? I've seen something similar for Kubeflow's training operators using <https://github.com/volcano-sh/volcano> and was wondering if it'd be possible to achieve for Flyte workflows.

For context, we'd like to create user-specific projects and enforce resource quotas on them in K8s; however, if we do this, we'd like to try to avoid multiple queued workflows contending for resources and let pods associated w/ a specific workflow get scheduled together. Another example of this would be <https://github.com/palantir/k8s-spark-scheduler>

And you will have to use kube batch or yunikorn etc

Example - array <https://docs.flyte.org/en/latest/deployment/cluster_config/flytepropeller_config.html|https://docs.flyte.org/en/latest/deployment/cluster_config/flytepropeller_config.html>

Ahh that's great. Thank you for the references <@UNZB4NW3S>