@ancient-wolf-19325 Is there an upper-bound to the number of tasks Flyte can schedule in parallel? I’m load testing a workflow that has max-parallelism set to 100K. When running this workflow I only actually see about 10K tasks scheduled at any given time at a time. I’m wondering if there is a global limit on the number of tasks that can be run at a time, or another config option available that might be interfering with maximum parallelization

Yes, there is an upper bound to the number of tasks Flyte can schedule in parallel, and it is influenced by several factors. While you can set max_parallelism at the workflow level (default is 25, but can be increased), FlytePropeller and the underlying Kubernetes cluster impose practical and sometimes hard-coded limits. For example, dynamic nodes can yield up to 4,000 nodes, and issues have been reported when attempting to fan out to 10,000+ tasks, often due to system or plugin constraints. Additionally, cluster resource limits, etcd storage, and FlytePropeller's internal handling can further restrict parallelism, regardless of the max_parallelism setting in your workflow. There is no single global hard limit documented, but practical scaling is typically in the low tens of thousands of concurrent tasks, depending on deployment and configuration specifics. See discussions in cc7362a2, 7e1e62ff, and 5a4561ec. Would you like more detail on how to tune these limits or on FlytePropeller/Kubernetes configuration for large-scale parallelism? Sources: - Flyte documentation: Nested parallelization - cc7362a2 - 7e1e62ff 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.