https://flyte.org
Join Slack
Hi! Using the flyte binary deployment on k8s, What is the difference between <plugins.workqueue.conf...
# flyte-support
f
Hi! Using the flyte binary deployment on k8s, What is the difference between plugins.workqueue.config.workers and propellor.workers ? Using this configuration, i only see 20 task pods at a time in k8s:
Copy code
configuration:
...
  inline:
    plugins:
      workqueue:
        config:
          workers: 500
    propeller:
      workers: 500
...
f
For one workflow ?
f
Correct. After a restart I see now 25 at a time. Requests for CPU: 4 No limits. I'm running only one workflow with 1000 jobs that should all run in parallel
Is flyte built in a way that no new pods will be submitted if no nodes are able to accommodate them? We are in EKS with auto scaling (karpenter) and I don't see any pod that is pending with no machine, which is odd. I expect to see 100s of pods pending waiting for a machine. Instead I see 25 at a time, which are assigned to an available machine right away.
f
25 is the default max parallelism
It’s let workflow you can override it at a global Level or per execution directly through the cli or ui
Copy code
pyflyte run --max-parallelism=100 ...
this will do it
it scales to as much as it can, but by default starts with conservative config
f
Got it. So i am looking to increase (maybe) the max parallelism per workflow. Thanks for pointing this out.
2 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by