<#6475 Add workflow concurrency control> Pull requ...
# flyte-github
c
#6475 Add workflow concurrency control Pull request opened by thomasjhuang ## Tracking issue This solves #5659 and also closes out #6309. The related flytekit change is here - flyteorg/flytekit#3267. Docs are added in this PR - unionai/unionai-docs#417 ## Why are the changes needed? This work is to provide workflow concurrency control at a project level. It is a barebones implementation with the guarantee that at most
x
many workflows will be running, but it cannot guarantee that
x
many workflows are always running. We can control concurrency across all versions, and users specify the controls via
LaunchPlan
instantiation:
Copy code
concurrency_limited_lp = LaunchPlan.get_or_create(
    name="my_concurrent_lp",
    workflow=my_workflow,
    concurrency=ConcurrencyPolicy(
        max_concurrency=3,
        behavior=ConcurrencyLimitBehavior.SKIP,
    ),
)
## What changes were proposed in this pull request? The primary mechanism is fairly straightforward - whenever we attempt to launch an execution, make a db query to check for running executions given the NamedEntityIdentifier triplet (project/domain/wf_name), and if the running executions is above the threshold for concurrency (
max_concurrency
) then we immediately fail to create the execution. ## How was this patch tested? Unit tests added to execution_manager_test.go, but namely this was internally tested at LinkedIn since we are porting this feature externally. More testing is in progress on local sandbox. ### Labels • added:
ConcurrencyPolicy
and logic surrounding workflow concurrency management, as well as db migration to add an index on
executions
table for
execution_phase
. ### Setup process ### Screenshots I was able to confirm locally that we receive the correct error message when we are running multiple workflows concurrently at limits of 1 and 2. Within the same project, running a workflow without limits works.
Copy code
Error: rpc error: code = ResourceExhausted desc = Concurrency limit (1) reached for launch plan concurrency_
limit_1. Skipping execution.
{"json":{},"level":"error","msg":"rpc error: code = ResourceExhausted desc = Concurrency limit (1) reached f
or launch plan concurrency_limit_1. Skipping execution.","ts":"2025-07-07T23:12:38-07:00"}
## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs #5659 and #6309, will also add in flytekit PR as reference. ## Docs link • Will add in docs https://bito.ai#summarystart| ### Summary by Bito This pull request introduces a feature for managing workflow concurrency at the project level, allowing users to set limits on concurrent executions via a ConcurrencyPolicy. It includes logic to check running executions, enhances logging, and updates the database schema to support these changes, addressing issues #5659 and #6309. flyteorg/flyte