#2578 [Core feature] Introduce configuration for max number of nodes per workflow mutation Issue created by honnix Motivation: Why do you think this is important? Today we have

max-parallelism

to control how many nodes will be started (asynchronously) per workflow mutation iteration. But in case of synchronous node this has no impact, because when node execution finishes, the state will be set to

success

that is a terminal state, which will not result in increment of parallelism; and the result is a worker hogging on a workflow. This could be solve by turning node execution into asynchronous, but there are many cases a node execution cannot or does not make sense to be asynchronous, for example a node simply does a string processing in memory. For those cases, if the totally number of worker is small and there exists a few extremely big and heavy fan-out workflows, it is easy to have an unfair scheduling result where those workflows will keep all the workers busy while other small workflows will be waiting for long. So it seems reasonable to have control of the max number of nodes being executed per workflow mutation iteration. Goal: What should the final outcome look like, ideally? This could either be achieved by changing semantics of

max-parallelism

for simplicity or introducing a new configuration for separation of concern. Describe alternatives you've considered We have considered turning some of the node executions into asynchronous, but there are node execution cannot or doesn't make sense to be asynchronous. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2578 [Core feature] Introduce configuration for max number of nodes per workflow mutation Issue closed as not planned by github-actions[bot] flyteorg/flyte