https://flyte.org logo
#ask-ai
Title
# ask-ai
g

Greg Linklater

02/12/2024, 1:12 PM
@Glime how can I run a subworkflow from a workflow in another file?
g

Glime

02/12/2024, 1:12 PM
To run a subworkflow from a workflow located in another file, you can utilize the concept of external workflows in Flyte. This involves defining a launch plan for the external workflow and then invoking this launch plan from within your main workflow. Here's a step-by-step guide based on the information provided in the context: 1. Define a Launch Plan for the External Workflow: First, ensure that the external workflow you want to run as a subworkflow has a defined launch plan. A launch plan in Flyte is a way to execute workflows with specific inputs and configurations. You can define a launch plan using the
LaunchPlan.get_or_create
method.
Copy code
from flytekit import LaunchPlan      # Assuming the external workflow is named 'external_workflow' and is defined in another file     external_workflow_launch_plan = LaunchPlan.get_or_create(         workflow=external_workflow,         name="external_workflow_launch_plan_name",         default_inputs={"input1": default_value1, "input2": default_value2}     )
2. Invoke the Launch Plan from Your Main Workflow: Once you have the launch plan for the external workflow, you can invoke it from within your main workflow as if you were calling a regular task. This is done by simply calling the launch plan object within the workflow definition.
Copy code
from flytekit import workflow      @workflow     def main_workflow(input1: InputType, input2: InputType) -> OutputType:         # Invoke the launch plan of the external workflow         result = external_workflow_launch_plan(input1=input1, input2=input2)         return result
3. Run Your Main Workflow: With the main workflow defined, you can now run it either locally or deploy it to a Flyte cluster. The external workflow will be executed as part of the main workflow execution. This approach allows you to modularize your workflows and reuse existing workflows across different projects or files without duplicating code. It's important to ensure that the external workflow and its launch plan are correctly registered in the Flyte backend before invoking them from another workflow. For more detailed examples and documentation on working with subworkflows and launch plans in Flyte, you can refer to the Flyte documentation and cookbook examples provided in the context.
k

Ketan (kumare3)

02/12/2024, 3:42 PM
You can simply import and run, but if you want to use pyflyte run then use the —copy-all flag for copying all files
Launchplan is another option
g

Greg Linklater

02/13/2024, 12:37 PM
@Ketan (kumare3) that is very helpful thanks!
@Ketan (kumare3) is there an equivalent to
--copy-all
for the
pyflyte package
command? If it is not necessary then could you please explain what is going on in the background?
k

Ketan (kumare3)

02/14/2024, 12:44 AM
ohh yes sorry for the delay
pyflyte run is inteded for scripts - fast iteration
package should zip up all the
files, workflows, tasks
into a self-contained tarball
when you say --copy-all - pyflyte run simply tars up everything it sees into a deployable ball
the way pyflyte run copies your script over is through a secure signed url that is generated by the server matching a hash value. the script is uploaded here. but if you reference modules and functions outside of the current script, then pyflyte cannot simply copy the file, this is where --copy-all instructs it to copy everything over. honestly it would be possible for pyflyte to figure this out magically and that is on the roadmap 🙂
g

Greg Linklater

02/14/2024, 6:48 PM
Fair enough. Thanks for the clarification
s

Scott Blackwood

02/15/2024, 4:32 PM
Similarly: @Glime How do I fetch workflows which are registered on a flyte cluster and run them from a parent workflow?
g

Glime

02/15/2024, 4:33 PM
To fetch workflows that are registered on a Flyte cluster and run them from a parent workflow, you can use the FlyteRemote functionality provided by
flytekit
. This allows you to interact with workflows that are registered in a Flyte cluster from your local environment or another workflow. Here's a step-by-step guide on how you can achieve this: 1. Set Up FlyteRemote: First, you need to set up
FlyteRemote
to interact with the Flyte cluster. This requires specifying the configuration details of the cluster, including the project, domain, and other necessary configurations.
Copy code
from flytekit.remote import FlyteRemote     from flytekit.configuration import Config      remote = FlyteRemote(         config=Config.auto(config_file="config.yaml"),         default_project="your_project_name",         default_domain="development",     )
2. Fetch the Registered Workflow: Use the
fetch_workflow
method of
FlyteRemote
to fetch the workflow you want to run. You need to provide the project, domain, name, and version of the workflow.
Copy code
workflow_to_run = remote.fetch_workflow(         project="your_project_name",         domain="development",         name="your_workflow_name",         version="your_workflow_version"     )
3. Execute the Workflow: After fetching the workflow, you can execute it by calling the
execute
method. You need to provide the necessary inputs for the workflow.
Copy code
execution = remote.execute(         workflow=workflow_to_run,         inputs={"input1": value1, "input2": value2},         wait=True     )
4. Access Execution Outputs: Once the execution is complete, you can access the outputs using the
fetch_outputs
method.
Copy code
outputs = remote.fetch_outputs(execution)
This approach allows you to dynamically fetch and execute workflows that are registered in a Flyte cluster from within another workflow. It's particularly useful for scenarios where you need to orchestrate complex workflows that depend on other workflows or when you want to modularize your workflow execution. Please note, the provided code snippets are for illustrative purposes. You may need to adjust them based on your specific requirements, such as the inputs and outputs of the workflows you are working with. For more detailed information and examples, refer to the Flyte documentation and the
flytekit
API reference.