#6309 [WIP] Add workflow concurrency controller
Pull request opened by
thomasjhuang
## DRAFT
This work is currently in progress, some portion of this work is incomplete and still being tested.
## Tracking issue
Flyte backend implementation of
#5659.
## Why are the changes needed?
This implements a flyte concurrency manager, which allows for granular control of concurrent workflows at a LaunchPlan version level. The concurrency controller is responsible for enforcing concurrency limits defined in launch plans, ensuring that the system doesn't exceed resource constraints and providing better predictability for workflow execution.
## What changes were proposed in this pull request?
## How was this patch tested?
The infrastructure changes have been tested in the Flyte sandbox environment to ensure the deployment is created correctly.
Full implementation testing is pending completion of the concurrency controller logic.
### Labels
•
added: For new features.
### Setup process
To test the concurrency controller in simulation mode:
1. Deploy Flyte with the updated charts
2. The concurrency controller will be enabled by default in simulation mode
3. Check the logs of the flyte-flyteconcurrency pod to verify it's reporting the correct configuration
To disable the simulation:
1. Set
concurrency_controller.enabled: false
in your values.yaml
### Screenshots
[None provided yet]
## Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
## Related PRs
## Docs link
### Summary by Bito
This pull request enhances concurrency management with granular workflow control capabilities at the LaunchPlan version level, introducing a concurrency controller for Flyte. It adds a new Cluster Resource Controller for dynamic Kubernetes resource management, updates scheduler policy definitions, and includes revised Helm charts and configuration files for both sandbox and production environments.
Unit tests added: False
Estimated effort to review (1-5, lower is better): 5
flyteorg/flyte