hi! how would Flyte most easily support multi-cluster/multi-cloud and compute allocation. I see discussion about volcano plugin here. What's the general direction? cc @freezing-airport-6809 @ancient-apple-95774

@silly-toddler-37820 I think you have some thoughts on this on Union.

would like to hear about self-hosted OSS solution too 🙂

We run multi-cluster with a custom Armada plugin. I've been meaning to open source some of it one of these days

Hey guys! In v2, the relationship between flyte entities (organizations, domains, project-domains, etc) is orthogonal and explicit with respect to clusters. Queues (new entity type) allow platform operators to determine which cluster a given action, task/trigger, task environment, project-domain, domain, or whole organization will execute on. So you can have: • my_org ◦ default_queue (priority 50, sends work to development cluster) ◦ high_priority queue (priority 90, sends work to development cluster) ◦ production domain ▪︎ production queue (priority 90, sends work to production cluster) In this setup, actions sent to development or staging domains will execute on the development cluster, with high_priority queue emptying before any actions from default_queue. Actions sent to production domain will execute on the production cluster. All of the scheduling logic is in the queue service, which understands all actions and queues at once and can make decisions like which queue should send to the executor in a given cluster. We will also look into resource-aware scheduling and preemption to better enforce priority in the coming months. @handsome-lock-30336 with respect to open source vs closed source, we haven't released OSS backend yet for F2. We are committing to parity between the systems so any code you write in U2 will be fully portable to F2 once we get the backend out. For now, U2/F2 are are in this closed beta so we can nail the experience and also make use of limited engineering resources.

We will also look into resource-aware scheduling and preemption to better enforce priority in the coming months

can you comment on how flyte-native resource-aware scheduling will be better than plugin-approach that community has adopted?

I don't necessarily think that they are mutually exclusive, for instance maybe we can use a plugin to perform the resource-aware scheduling in the control plane (where we have knowledge of all actions, queues and clusters). I haven't looked that deeply at it - just thinking about the above topology as a way of enabling the key user flows we have been hearing about: • allow tasks within the same workflow to execute on different clusters • allow priority to be specified such that low-priority jobs (backfill) will not block high-priority jobs (MLE waiting on result) • attempt to schedule tasks on spot nodes across many regions and cancel all but the first action that successfully schedules • burst from a reserved to on-demand cluster based on utilization