Hi!
Using the flyte binary deployment on k8s,
What...
# flyte-supportf
fresh-jackal-53335
03/09/2025, 1:25 PMHi!
Using the flyte binary deployment on k8s,
What is the difference between plugins.workqueue.config.workers and propellor.workers ?
Using this configuration, i only see 20 task pods at a time in k8s:
Copy code
configuration:
...
inline:
plugins:
workqueue:
config:
workers: 500
propeller:
workers: 500
...f
freezing-airport-6809
03/10/2025, 3:00 PM
For one workflow ?
f
fresh-jackal-53335
03/10/2025, 4:53 PMCorrect. After a restart I see now 25 at a time.
Requests for CPU: 4
No limits.
I'm running only one workflow with 1000 jobs that should all run in parallel
fresh-jackal-53335
03/10/2025, 4:57 PMIs flyte built in a way that no new pods will be submitted if no nodes are able to accommodate them? We are in EKS with auto scaling (karpenter) and I don't see any pod that is pending with no machine, which is odd. I expect to see 100s of pods pending waiting for a machine. Instead I see 25 at a time, which are assigned to an available machine right away.
f
freezing-airport-6809
03/10/2025, 8:30 PM
25 is the default max parallelism
freezing-airport-6809
03/10/2025, 8:30 PM
It’s let workflow you can override it at a global
Level or per execution directly through the cli or ui
freezing-airport-6809
03/10/2025, 8:32 PM
Copy code
pyflyte run --max-parallelism=100 ...freezing-airport-6809
03/10/2025, 8:32 PM
this will do it
freezing-airport-6809
03/10/2025, 8:32 PM
it scales to as much as it can, but by default starts with conservative config
f
fresh-jackal-53335
03/11/2025, 6:21 AMGot it. So i am looking to increase (maybe) the max parallelism per workflow.
Thanks for pointing this out.
4 Views