@ancient-wolf-19325 Is there a way to limit a number of running tasks of particular type in parallel across workflows? Some kind of global semaphore? My use case is calling an API and need to limit concurrent requests to prevent quota depletion on the server side.

Flyte does not currently provide a built-in, global semaphore or direct mechanism to limit the number of running tasks of a particular type across all workflows. Task-level concurrency controls (such as limiting parallelism within a workflow or map task) are supported, but global cross-workflow throttling for a specific task type is not natively available. This limitation is discussed in community issues, and while there are workarounds (like implementing a "resource lock" task or using custom plugins), these are not first-class features and can lead to inefficiencies or dangling tasks if many executions are queued or blocked. There is ongoing discussion and RFCs about introducing more general resource locking or concurrency controls, but as of now, no official feature exists for this use case in Flyte's core platform or SDKs (source 1, source 2, source 3, source 4, source 5). Would you like more detail on possible workarounds or the current state of RFCs and proposals for this feature? Sources: - github.com/flyteorg/flyte/issues/5125 - github.com/flyteorg/flyte/pull/5659 - github.com/flyteorg/flyte/issues/267 - github.com/flyteorg/flyte/issues/872 - source 5 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Can you summarize the workarounds and if they were successful?

Workarounds discussed include implementing a "resource lock" task at the start of workflows, which queries Flyte for running executions and waits (e.g., with sleep loops) if the concurrency threshold is reached. This approach can throttle concurrent executions but is considered hacky and leads to many semi-dangling or blocked tasks if the number of queued executions is high, making it inefficient and not scalable (source). Another suggestion is to use distributed locks or external systems (like Zookeeper), but this adds operational complexity and external dependencies (source). No workaround is described as fully successful or robust for production-scale needs; all are acknowledged as suboptimal until a native feature is implemented. Would you like more detail on how to implement these workarounds or updates on the status of native support? Sources: - github.com/flyteorg/flyte/issues/5125 - github.com/flyteorg/flyte/issues/267 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.

There is a setting here, let me know if it works for your needs - https://www.union.ai/docs/v1/flyte/user-guide/core-concepts/launch-plans/concurrency-control