New RunLLM response generated: >>>*# Question* We...
# runllm-flyte-trial
a
New RunLLM response generated: >>>*# Question* We have a workflow that is composed (for now) of 2 parts - generic one and a custom one. I want to have a parameter to the top-level workflow that will determine which workflow is running after the generic part. Trying to put it into code:
Copy code
@dynamic
def do_next_step(next_step: str, generic_output: ...):
  # ...somehow get the reference launch-plan
  ref_lp = get_ref_lp(next_step)
  ref_lp()

@workflow
def primary(generic_args..., next_step: str):
  generic_output = generic_task(generic_args...)
  do_next_step(next_step, generic_output)
I managed to do something like that:
Copy code
@dynamic
def do_next(next_step_module: str, next_step_func: str) -> None:
    next_step_mod = importlib.import_module(next_step_module)
    next_step = getattr(next_step_mod, next_step_func)
    next_step()
where the module+func are ref plan, e.g.:
Copy code
@reference_launch_plan(
    project="flytesnacks",
    domain="development",
    name="data_types_and_io.file.normalize_csv_file",
    version="-ism2z-QiRgPisO-KY_YTA"
)
def normalize_csv_file(
        csv_url: FlyteFile,
        column_names: List[str],
        columns_to_normalize: List[str],
        output_location: str,
) -> FlyteFile:
    ...
That worked but means that I have to maintain launch plans for each sync next step. Essentially, there's no reason I shouldn't be able to get the latest launch plan in runtime using Flyte client which is part of the current context. But I didn't find a way to convert the Flyte API object to something a dynamic workflow can trigger. Another thing I tried is figuring out how to trigger a workflow from within a task. I can then "wait" on the workflow to finish. That's less ideal but also an option. Lastly, I figured we can use a different approach, and generate the workflow dynamically using imperative workflow and register it if needed, then run it. I hope I'm making sense. In more simple terms - I'm trying to determine and execute next steps in a workflow (dynamic or otherwise) according to input arguments. # Answer To determine and execute the next steps in a Flyte workflow according to input arguments, you can use
reference_launch_plan
and Flyte's dynamic workflows. This approach allows you to dynamically select and run a launch plan based on input parameters. Here's a step-by-step breakdown: 1. Define Reference Launch Plans: Predefine the launch plans you want to dynamically select. 2. Dynamic Workflow: Implement a dynamic workflow to select and execute the appropriate launch plan based on input arguments. 3. Primary Workflow: Use the dynamic workflow within your primary workflow to handle the conditional execution logic. Here's an example code implementation: Step 1: Define Reference Launch Plans Define the reference launch plans for the workflows you want to dynamically select.
Copy code
from flytekit import reference_launch_plan, workflow, dynamic

@reference_launch_plan(
>>```
project="flytesnacks", domain="development", name="data_types_and_io.file.normalize_csv_file", version="latest" ) def normalize_csv_file( csv_url: FlyteFile, column_names: List[str], columns_to_normalize: List[str], output_location: str, ) -> FlyteFile: ...
Copy code
*Step 2: Implement Dynamic Workflow*

Create a dynamic workflow that selects and executes the appropriate launch plan.
from flytekit import task, workflow, dynamic @dynamic def do_next_step(next_step: str, generic_output: ...): if next_step == "normalize_csv": ref_lp = normalize_csv_file else: raise ValueError(f"Unknown next step: {next_step}") ref_lp(csv_url=generic_output['csv_url'], column_names=generic_output['column_names'], columns_to_normalize=generic_output['columns_to_normalize'], output_location=generic_output['output_location'])
Copy code
*Step 3: Primary Workflow*

Use the dynamic workflow within your primary workflow to handle the conditional execution logic.
@workflow def primary(generic_args..., next_step: str): generic_output = generic_task(generic_args...) do_next_step(next_step=next_step, generic_output=generic_output) ``` Notes 1. Reference Launch Plans: Ensure that the reference launch plans are correctly registered in Flyte. 2. Dynamic Workflow: The dynamic workflow (
do_next_step
) should handle the logic to select and execute the appropriate launch plan. 3. Primary Workflow: The primary workflow (
primary
) should call the dynamic workflow with the necessary arguments. This approach allows you to dynamically select and execute different workflows based on input parameters, leveraging Flyte's dynamic workflows and reference launch plans. Would you like more details on any specific part of this implementation? Sources: - Flyte Reference Launch Plan - Flyte Dynamic Workflows Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1719507212.124169 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.