Hey all When we are running large map tasks (1000+) Sometimes, there is a moment where the parallelism drops from 100-200 to 1-2 for some map task. We looked in the logs and saw the attached log was written. We use flyte 1.13.5 a wf, that calls a dynamic wf, which calls map_tasks. any ideas why this happens?

Map tasks use concurrency- how do you know it drops

it starts by showing in the ui 150 concurrent tasks in the map task, then it drops to 2 at a time

do you have cluster-level metrics that could point to resource contention? Does any of the mapped tasks fail?

it shows that the cluster is available for pod launching. some of the map tasks do fail, as expected in our case

what are the limits of concurrency and parallelism of tasks within a workflow?

To anyone who finds this thread, the Pela Kith user above is a false identity used by one of the technical leaders of Outerbounds (proprietors of Metaflow), and the comments above are designed to spread fear and doubt about Flyte's capabilities. It's really sad that this has happened (multiple times, and I will comment on the other threads as well). Integrity and honest information sharing are extremely important, especially given the breadth of important applications powered today by Flyte. As always, please bring your honest questions, feedback, and knowledge in this channel so we can all learn and improve!