Hi! I have LaunchPlan created in Python that I wan...
# ask-the-community
m
Hi! I have LaunchPlan created in Python that I want to trigger in my CI/CD flow based on file changes. The documentation shows how to create an execution given a spec file, but how do I actually execute a launch plan that’s already created programmatically?
k
You can use flytekit.remote
m
Ok thanks. It’s not possible to use any of the CLI tools?
k
Hmm you can use flytectl
m
Can you give an example? I know how to get launch plans and create executions, but execution files are clunky for CI/CD. I basically just want to execute a launch plan with some dynamic inputs. I could just execute the workflow directly but launch plans have some other nice benefits, like notifications.
y
flytectl is just pulling the default launch plan btw for the workflow.
it’s not creating a new launch plan
unf the spec file is the way. the yaml file is a bit janky yeah but it does allow the user to express more complex types.
we should add this to pyflyte ketan?
k
I think so
can you file an issue
this is not a hard add
m
Maybe I’m not using the Flyte API the correct way. My use case is the following: I am monitoring a folder of metadata files for different processes. Whenever these change, I want to trigger a Workflow with the changed process as input. This all works in my GitHub action, but it would be nice to execute a pre-existing LaunchPlan instead of a Workflow. Does this make sense?
I can add file an issue if this sounds like a reasonable use-case. Would it go into
pyflyte
or
flytectl
or both?
h
So the question here is that we have one model architecture that we want to train and fine tune for many different datasets, lets say we have N of them. Ideally, we would like to have one workflow that produces a model, use N different launch plans, one for each dataset, to trigger a specific training for each model. The launch plan it self could be the same for each dataset, except for one, or many, parameters that should vary based on dataset. Similar to what you do with overriding values in Helm installations for instance. How do we structure this? We just want to define the workflow once in code, and then use variables to train it for a specific dataset. Let me know if it is not clear and I can explain further 🙂
k
@Martin Hwasser / @Hampus Rosvall mostly makes sense, But some confusion- let me paraphrase- You want to trigger a workflows with different dataset? If so why can you not just use different input values. Launchplans are useful if you want to change the interface of a workflow- default some inputs and or schedule them- as scheduling can only provide time as an input
h
Copy code
If so why can you not just use different input values.
How would you then do that? I think the confusion we have is that we don’t really know how to achieve different input values for a workflow in an automated way e.g., in CI/CD workflow. In the UI it is trivial
k
Ohh you just don’t know how to send different inputs?
Like pyflyte run —dataset x Pyflyte run — dataset y ? Like that Flytekit.remote cab make this all programmable
m
No,
pyflyte
can’t run with
LaunchPlan
, at least it’s not documented
I think @Hampus Rosvall meant to write “different input values to a LaunchPlan”, not workflow.
We could of course use python and flytekit.remote but it’s puzzling to me that it’s not possible to do with the CLI tools.
e
just following up on this, I opened https://github.com/flyteorg/flyte/issues/2998 to track the work to enable launchplans in pyflyte run. Should be a pretty simple contribution. 🙂
102 Views