Currently our GCP account has the ability to support about 2500-3000 tasks in parallel based on the CPU/MEM requirements of these Flyte tasks Initially the workflow started and FlytePropeller scheduled and ran about 500 tasks in parallel. I figured it would ramp up over time as new nodes were provisioned However after about 30 minutes the concurrent pods dropped to around 150 tasks and it’s been that way ever since (over 60 minutes) We have about 40 nodes running currently and they are only using about 25% of CPU/Mem, so not sure why FlytePropeller isn’t scheduling more tasks I also checked the CPU/Mem quota in the namespace and it’s well under the limit as well

cc: @Justin Tyberg

Having a diagram of the fan out will be interesting to see

the Flyte Console graph for the workflow is not very interesting

If so why not use map task?

I think there are a few things over here- max parallelism, number of workers, k8s config etc

I can launch as one workflow execution per file if that would help, but I’d like to understand the config options available to our deployment like you said max parallelism, etc.

Currently we only have 1 FlytePropeller pod that has 1CPU and 2GB mem it doesn’t seem to be maxing out on either. Propeller also has 50 workers configured

A VC

One draft running 120 8000 tasks I don’t think it is supported

❤️