Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hey Flyte team, happy Friday. When it comes to the maxParallelism config where should I be looking for errors as i start to increase this number?

I guess i should ask where would we look for limitations in the system while we start to scale up the Parallelism?

Maxparallelism by definition
```	// Controls the maximum number of task nodes that can be run in parallel for the entire workflow.
	// This is useful to achieve fairness.```
So if we start increasing parallelism for one of the workflows then it would impact fairness for other workflows which have a lesser value since propeller would be doing more work for ones which have higher parallelism . If all have the same increased parallelism then it increases the burden on propeller . I would have Ketan comment more on this

<@UPTRGR537> <@USU6W5ATA> follow up question from <@U03BSHYHQ1K> if you could revisit this
&gt; when is this used only for map tasks? Also why not just slam 1000 in for it.

Adding <@U029U35LRDJ> to comment on the map tasks and impact with maxParallelism with 1000 of tasks .

so map tasks actually set their own parallelism, meaning they execute without respecting the max parallelism set on the workflow. for example, a workflow with max parallelism 2 will run 2 tasks simultaneously, if one of those is a map task that has set 100 max parallism when that task executes it will execute 100 in parallel. changing this value on map tasks will have the same effect as changing it on the workflow, namely a larger number of concurrent task executions.

While we are running a very large dynamic it seems to be stuck moving higher than 25 concurrent pods. I assumed this was the maxParallelism, but ive edited the value above 25 and I have not seen any movement. Does this actually have nothing to do with maxParallelism?

I did edit this while the job was running.

This is what the scale looks like. I just dont know what knob to pull on to increase this limit.

How did you edit it while the job was running? I think the only ways to modify this value for an execution are to set in on the top level `@workflow` decorator or pass it into the execution context through FlyteConsole.

Oh sure, yeah so that will not be picked up while a workflow is executing. Basically, that value is set during workflow construction. So once the workflow starts it has a max parallelism that cannot be modified for the entire execution.