How does Flyte handle overloaded kubernetes clusters. For example, if the cluster has space for 200 tasks but we have 2000 runnable tasks, what will happen?

It seems likely that Flyte relies on Kubernetes to manage task scheduling and resource allocation. When a Kubernetes cluster is overloaded, tasks that cannot be scheduled due to resource constraints will remain in a "Pending" state until resources become available. According to this discussion, you can limit the number of concurrent tasks by setting a limit on the autoscaling of your node pool or node group. This means that once the limit is reached, additional tasks will wait until resources are freed up. My understanding is that Flyte does not natively limit the number of executions per project or workflow, but you can manage this by configuring Kubernetes appropriately. If you have a large number of tasks, it might be beneficial to use a multi-cluster setup to distribute the load, as suggested in the Flyte documentation. Would you like more detail on how to configure Kubernetes or Flyte for handling such scenarios? Sources: - Flyte Discussion on Task Execution Limits 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.