https://flyte.org logo
#ask-the-community
Title
# ask-the-community
e

Eduardo Matus

11/02/2023, 4:12 PM
Hi everyone… Using Dynamic workflows.. is there a way to control the parallel running workflows? currently a process is launching like 100 workflows but only a 10% of those are actually running (although the UI says all of them are running) because of project cpu and mem quotas. Is there a way to put the 90% remaining workflows in queue or scheduled state?
d

Dan Farrell

11/02/2023, 4:25 PM
This isn't a direct answer to your question, but it sounds like you want to update your task-resource-attribute and your cluster-resource-attribute using this? https://docs.flyte.org/en/latest/deployment/configuration/general.html#task-resources
r

Rahul Mehta

11/02/2023, 4:35 PM
Pretty sure this is related to my earlier question too https://flyte-org.slack.com/archives/C01P3B761A6/p1698940470184879
Curious if there's a hidden default setting that's limiting our concurrency here
d

Dan Farrell

11/02/2023, 4:44 PM
fwiw I had some limit problems, but they were removed after I set my tra and cra. I have a pr to update the docs for the demo cluster here https://github.com/flyteorg/flytesnacks/pull/1183/files
k

Ketan (kumare3)

11/03/2023, 2:19 AM
You can set max parallelism and that will control it
You can set it at execution time
r

Rahul Mehta

11/03/2023, 2:19 AM
Applied both the cra & maxParallelism changes and looks like things are chugging along now. We've got some tuning to do there but we just set this at the cluster level
Might need to scale out our propeller now since we're running ~20 concurrent large backtests (which fan out to >10 subworkflows for individual model training)
But @Dan Farrell @Ketan (kumare3) thanks for the help!
k

Ketan (kumare3)

11/03/2023, 2:21 AM
I do t think so - I think you might just increase propeller memory and add more workers
Propeller scales pretty well
r

Rahul Mehta

11/03/2023, 2:21 AM
Right, not necessarily bumping replicas or anything like that/sharding, we're still at default # of workers (40)
k

Ketan (kumare3)

11/03/2023, 2:25 AM
Ohh ya
You also should bump kube api limits - client
Check the docs
r

Rahul Mehta

11/03/2023, 2:40 AM
Kk. Is Flyte on modern k8s client-go versions? The ones >= 1.25 have much better support for clusters with many resources/CRDs
k

Ketan (kumare3)

11/03/2023, 2:29 PM
Yup