Let’s say we have a workflow that should be associ...
# ask-the-community
h
Let’s say we have a workflow that should be associated with N different inputs, entailing N different launch plans. How would you structure/manage this in code? We have adopted an idea of managing the launch plans in a separate folder, next to the flyte workflows and thus deploying the launch plans and workflow in separate - see example below. Does this make sense for the N:1 relationship between launch plans and workflows? This entails a lot of duplication of boilerplate code in
lp.py
but it is maybe the cleanest solution to this problem?
Copy code
(.venv) ☁  flyte [main] tree .
.
├── <http://in_container.mk|in_container.mk>
├── launchplans
│   └── lp.py
└── workflows
    ├── wf.py
h
I’m probably missing something, what is the boilerplate that ends up getting duplicated?
h
Oh right, I forgot to add that code snippet. So
lp.py
then is
Copy code
# imports

lp_definition_use_case_1
...
lp_definition_use_case_N
If I change launchplan_i, then I would need to re-deploy all launch plans. But maybe that’s fine?
And is there a way to deploy the launch plans standalone, if I am managing the launch plans in
lp.py
and want to register only the launchplans, I should not do so with e.g.,
pyflyte register
since it is aimed for tasks and workflows as it requires an image?
s
@Hampus Rosvall, you can use
pyflyte register
for
lp.py
as well:
pyflyte register lp.py
, without sending an
--image
argument if you’re okay with using the latest flytekit image.
If I change launchplan_i, then I would need to re-deploy all launch plans. But maybe that’s fine?
I believe that’s the right path for now. @Eduardo Apolinario (eapolinario) @Yee, any ideas?
h
@Samhita Alla I have already deployed my workflow using
--image
flag. What does it entail to deploy launch plans using the latest flytekit image then?
s
I don’t think an
--image
flag is required cause a launch plan executes a workflow of specific version which might have already been associated with an image (in your case).
h
Gotcha, thanks for your help! So it makes sense then?
s
Yeah!
h
The launch plan definition below deploys a new launch plan, but also the entire workflow using
pyflyte register flyte/launchplans/lp.py
- is there a way to only deploy a launch plan that runs the latest version of a workflow without deploying a new workflow version? What is best practice in terms of working with launch plans?
Copy code
from flytekit import LaunchPlan
from ..workflows import wf

LaunchPlan.get_or_create(
    workflow=<http://wf.wf|wf.wf>,
    name="flytesnacks_lp",
    default_inputs={},
)
Ideally I would like to be able to deploy new launch plans that references the latest version of some workflow, without manually editing the version in e.g.,
launch_plan.yaml
@Samhita Alla in case you didn’t see. Basically I am interested in when to deploy a new version of a launch plan. Should I just do that every time I deploy the workflow?
s
The workflow’s getting registered cause you’re importing it. I’m not sure how to register only a launch plan. @Yee @Eduardo Apolinario (eapolinario), is that possible today?
h
I am just after some guidelines in terms of how I should think about working with launch plans. To me it seems like it is considered best practice to register launch plans whenever you register a workflow. If that is the preferred way then that is all good! 🙂
s
Yeah! That seems like a better way, although you should be able to register a launch plan without registering a workflow. I think this should be possible through FlyteRemote’s `register_launch_plan`: https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#registering-entities; I’m not pretty sure, though.
But I’d recommend you to register launch plans whenever you register workflows for now. I’ll talk to my team about registering only launch plans and get back to you.
Also, launch plan accepts workflow as an argument. I guess it makes total sense to register a workflow along with a launch plan so that the launch plan is associated with the latest workflow version. There’s no version argument in
LaunchPlan.get_or_create
for you to associate your launch plan with a particular workflow cause we aren’t giving an already registered workflow as an input to it. Does this make sense?
156 Views