Hey Flyte team, happy Friday. When it comes to the maxParallelism config where should I be looking for errors as i start to increase this number?

cc @Prafulla Mahindrakar / @Kevin Su

What errors?

I guess i should ask where would we look for limitations in the system while we start to scale up the Parallelism?

Maxparallelism by definition

// Controls the maximum number of task nodes that can be run in parallel for the entire workflow.
	// This is useful to achieve fairness.

So if we start increasing parallelism for one of the workflows then it would impact fairness for other workflows which have a lesser value since propeller would be doing more work for ones which have higher parallelism . If all have the same increased parallelism then it increases the burden on propeller . I would have Ketan comment more on this

@Prafulla Mahindrakar @Kevin Su follow up question from @Louis DiNatale if you could revisit this

when is this used only for map tasks? Also why not just slam 1000 in for it.

Adding @Dan Rammer (hamersaw) to comment on the map tasks and impact with maxParallelism with 1000 of tasks .

so map tasks actually set their own parallelism, meaning they execute without respecting the max parallelism set on the workflow. for example, a workflow with max parallelism 2 will run 2 tasks simultaneously, if one of those is a map task that has set 100 max parallism when that task executes it will execute 100 in parallel. changing this value on map tasks will have the same effect as changing it on the workflow, namely a larger number of concurrent task executions.

While we are running a very large dynamic it seems to be stuck moving higher than 25 concurrent pods. I assumed this was the maxParallelism, but ive edited the value above 25 and I have not seen any movement. Does this actually have nothing to do with maxParallelism?

How did you edit it while the job was running? I think the only ways to modify this value for an execution are to set in on the top level

@workflow

decorator or pass it into the execution context through FlyteConsole.

Ah ok, I edited the adminconfig

Oh sure, yeah so that will not be picked up while a workflow is executing. Basically, that value is set during workflow construction. So once the workflow starts it has a max parallelism that cannot be modified for the entire execution.

Perfect, thank you so much… AGAIN

no problem! 😄