I have a single workflow that runs with 12 differe...
# ask-the-community
g
I have a single workflow that runs with 12 different launch plans in a project. I'd like them to run sequentially in the evening, since they interact intensively with external resources. Is there a recommended way to do this?
Or maybe just a concurrent workflow limit that can be applied per project?
j
you could use chaining https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/chain_entities.html#chain-flyte-entities maybe chain half of the workflow to run after the first half, i wonder if you could also specify the max parallelization parameter to low so that things dont get run all at the same time
g
Hey Jay, kind of you to reply, thank you. Right now, each workflow runs great on its own. I just don't want 2 workflows running at the same time. So I might be able to use subflows, as you mentioned --if I was clever about how I constructed things. To your second point, Setting the max parallelization seems like the general case solution to me, but I don't know how to do it. I'd need to limit the total number of running workflows on a project basis (max workers?). The cluster resources page looks promising, but I don't yet understand how to use it. https://docs.flyte.org/en/latest/deployment/configuration/general.html#:~:text=Cluster%20Resources,want%20to%20see%20filled%20in.
j
you can set the max parallelization when you set up a launchplan like this
LaunchPlan.create("my LP", my_LP, max_parallelism=<some value>)
g
That's for limiting concurrent individual tasks, as I understand it. I'm trying to limit the concurrent workflows.
yeah i wonder if you can set it at your parent workflow that runs those 12 workflows
g
Wanted to report back -- subflows are working like a treat
j
That’s great to hear 🎉