https://flyte.org logo
#ask-the-community
Title
# ask-the-community
g

Garret Cook

01/16/2024, 6:10 PM
I have a single workflow that runs with 12 different launch plans in a project. I'd like them to run sequentially in the evening, since they interact intensively with external resources. Is there a recommended way to do this?
Or maybe just a concurrent workflow limit that can be applied per project?
j

Jay Ganbat

01/17/2024, 3:28 AM
you could use chaining https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/chain_entities.html#chain-flyte-entities maybe chain half of the workflow to run after the first half, i wonder if you could also specify the max parallelization parameter to low so that things dont get run all at the same time
g

Garret Cook

01/17/2024, 3:34 AM
Hey Jay, kind of you to reply, thank you. Right now, each workflow runs great on its own. I just don't want 2 workflows running at the same time. So I might be able to use subflows, as you mentioned --if I was clever about how I constructed things. To your second point, Setting the max parallelization seems like the general case solution to me, but I don't know how to do it. I'd need to limit the total number of running workflows on a project basis (max workers?). The cluster resources page looks promising, but I don't yet understand how to use it. https://docs.flyte.org/en/latest/deployment/configuration/general.html#:~:text=Cluster%20Resources,want%20to%20see%20filled%20in.
j

Jay Ganbat

01/17/2024, 3:38 AM
you can set the max parallelization when you set up a launchplan like this
LaunchPlan.create("my LP", my_LP, max_parallelism=<some value>)
g

Garret Cook

01/17/2024, 3:39 AM
That's for limiting concurrent individual tasks, as I understand it. I'm trying to limit the concurrent workflows.
yeah i wonder if you can set it at your parent workflow that runs those 12 workflows
g

Garret Cook

01/18/2024, 12:23 AM
Wanted to report back -- subflows are working like a treat
j

Jay Ganbat

01/18/2024, 12:49 AM
That’s great to hear 🎉