Hey everyone, looking into making my workflows more reliable by limiting how many executions can run concurrently. I am looking in the docs but haven't found a configuration that manages this. I'm ok with any parallelism within workflow executions My main goal is to handle peaks of incoming data (that trigger new workflow executions) better and prevent rising costs and jobs from failing due to timeouts and limited resources/failing to scale the cluster. Looking at https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/performance/#1-workers-the-workqueue-and-the-evaluation-loop and https://www.union.ai/docs/v1/flyte/deployment/configuration-reference/scheduler-config/#queue-configcompositequeueconfig

We setup Kueue to work with Flyte so that each team's gets their own ClusterQueue. When the ClusterQueue gets busy, Kueue acts as queue for the incoming workloads. https://kueue.sigs.k8s.io/docs/

@worried-airplane-87065 Awesome. I was also looking into that earlier, do you queue per task type? Per workflow execution? Does flyte set the status to running, if so how do you handle timeouts?

We have a queue per team with their own limits. We just inject "kueue.x-k8s.io/queue-name: MY_TEAM-local-queue" for workflow invocations. We see "Queued" when the workload is pending and then "Running" once it's running. I haven't looked into timeout behaviors yet.

The concurrency is added recently and I think flyte only support SKIP option for now. Now quite sure about the behavior, whether the task will be ignored or stayed in the Not Started state and can be restart

It does indeed only support Skip. I'll test it out and have a look if it fits my needs