https://flyte.org logo
Join Slack
Powered by
Hi everyone, we use reference launch plans to run ...
# flyte-support
l
Hi everyone, we use reference launch plans to run sub-workflows. Is there a way to set the maximum parallelism for sub-workflows when using reference launch plans? Thank you!
t
you mean override the parallelism that’s set on those subwf launch plans already?
c
@thankful-minister-83577 no. He means overriding the parallelism set for the parent workflow. It seems that launchplan parallelism, even if customized, has no effect when triggered as a sub-workflow and it takes the parallelism of the parent workflow
l
@curved-petabyte-84246 exactly
t
could you give me a more concrete example?
a repro-able example i mean. i suspect the answer is no so i will use it to make a gh issue.
c
We'll try to provide a minimally reproducible example. Thanks!
l
Hi @thankful-minister-83577. There is already an open gh issue for this: https://github.com/flyteorg/flyte/issues/5555.
@thankful-minister-83577 Do you have any suggestions for a workaround, or what we can do to resolve this issue? We need to trigger our sub-execution with a different parallelism setting than the main execution
t
i’ve been ooo sorry… can i think about this a bit more and get back to you?
discussed this with someone else internally, we do not currently think there is a workaround for this.
l
Sure! Thank you!
t
need a bit more context, will discuss internally some more.
some technical background on this… the reason it’s hard to just merge that pr that’s in the ticket is that however you do it, there’ll be requests from users to do the opposite. using your example, if we flip the subwf to use 1 instead of 100, we’ll break the people who are currently expecting it to be 100. We’re debating just allowing users to override. However the overriding behavior today is limited to TaskNodeOverrides. A launch plan node not only does not have task node overrides, it doesn’t have any overrides. We can add it of course, but lately there’s been enough requests for things to add that we were thinking of maybe taking a different approach.
not set on the syntax at all, but for situations like this, how would you feel about something like:
Copy code
@workflow
def parent():
  sub_launch_plan().clone_with(max_parallelism=10)
this would create a different launch plan, with a different name, overriding the max_parallelism field with 10. (side note - we’re going to alias “max_parallelism” to “concurrency” and alias them - the word concurrency makes more sense to us)
the same would apply for tasks. (and potentially workflows, though it’s not clear what one would replace on a workflow)
this would introduce a new concept of “copying” a task, rather than just setting an override on a task (or lp). For tasks, the name of the cloned task will likely be different. This means that caching will not be shared.
what do you guys think? cc @glamorous-carpet-83516 @high-accountant-32689
l
If it allows us to run our sub-workflows without inheriting the concurrency of the main workflow, then we're fine with that 🙂
h
I like this idea of 
clone_with
(and the point about parallelism -> concurrency). Are we going to track this in https://github.com/flyteorg/flyte/issues/5555?
c
@thankful-minister-83577 thanks for the insights and suggestion. To be honest, cloning a launch plan isn't ideal for us for few reasons: • Multiple launch plans are created - depending on the implementation, this could result in multiple launch plans "trashing" the system. We may do it in a dynamic workflow, but even if not, it will create a launch plan each "register" operation. • The launch plan of the sub-workflow is already configured with the desired concurrency. In terms of usage, I don't want the calling workflow to mess around with the already configured value. The calling workflow should treat the sub-workflow as a "black box" and not aware of the implementation details of the sub workflow requiring a different parallelism. In other words, I strongly believe that what described in #5555 is the "correct" behavior. While I understand there are different opinions, maybe we can offer a different flag asking Flyte to respect the configured value: 
sub_launch_plan().override(respect_concurrency=True)
. WDYT?
5 Views
Open in Slack
PreviousNext
Powered by