@ancient-wolf-19325 We have a Flyte setup with a number of different workflows, some dependent on each other. For now, we are building a separate image for each workflow and registering them one by one. When we want to run the whole pipeline we start the workflows one by one from the UI and monitor the progress. When successful, we can launch the next one. Now we want to start programmatically orchestrating the workflows. Our initial idea is to use flyte remote to start the latest production version of each wf and upon success, start the next one. This would be built into a higher level "orchestration" flyte workflow. Issue with this approach that we must use a lot of boilerplate to launch each subworkflow. Additionally, the level of monitoring on the orchestration workflow level is not great. To see what tasks are running etc one must find the subworkflow execution and monitor from there. We are wondering if there is a more "native" way of doing this. Some requirements for the setup • We must be able to use different images per workflow • Must be frictionless to develop the subworkflows as separate entities. Then upon release, use the latest version in the orchestration flow. Main issue we are running into: how do we serialize and register all of the subworkflows at the same time (so we can import them in the orchestration wf) and use different images for each (bc we have different requirements for each).

Based on Flyte’s design, the most native approach is to leverage subworkflows to compose your higher‐level orchestration workflow. This means you can: • Develop each workflow separately in its own module with its own container image (using the container_image parameter in the @task decorator or via the --image flag/configuration). • Organize your code so that when you run a multi‑package registration (for example, by running pyflyte register on your project directory), all of the workflows get serialized and registered together. This lets you import the subworkflows into your orchestration workflow and “call” them like ordinary Python functions. The resulting DAG will then combine all subworkflows, and the UI will show the full execution graph for easier monitoring. In practice, you’d use Flyte’s native subworkflow mechanism (described in the documentation on subworkflows) to have one workflow call another rather than launching them separately via FlyteRemote. That approach minimizes boilerplate: you can simply import and call the latest registered version of each subworkflow, and the individual container images remain encapsulated within their own registered workflows. This setup meets your requirements because: • Different images can be maintained per workflow through per‑task image configuration. • Development remains frictionless as each subworkflow is built and registered independently, and then brought together in the orchestration workflow. • The overall orchestration DAG is visible and monitorable from a single UI view. Would you like more details on how to set up multi‑package registration or on configuring different images for each workflow? --- Sources: - Registering workflows – Flyte documentation - Multiple images in a workflow – Flyte documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 "Would you like more details on how to set up multi‑package registration or on configuring different images for each workflow?" Yes please

Below is an outline of how you can set up multi‑package registration along with configuring different images for each workflow. Multi‑Package Registration You can organize your subworkflows in different directories (or packages) and register them together. For example, if you have subworkflow directories called “subwf1” and “subwf2,” you can register both at once using the

--pkgs

flag:

pyflyte --pkgs subwf1 --pkgs subwf2 register --image mindmeld="ghcr.io/your_org/subwf1-image:latest" --image borebuster="ghcr.io/your_org/subwf2-image:latest"

You can also specify images in your configuration file (e.g. 

$HOME/.flyte/config.yaml

) so that when you run the registration command, each subworkflow uses its designated image. Configuring Different Images in Workflows Inside each subworkflow code, when defining tasks you can pass a custom image using the

container_image

parameter in the

@task

decorator. For example:

from flytekit import task, workflow

@task(container_image="ghcr.io/your_org/subwf1-image:latest")
def task_subwf1() -> str:
    return "Result from subworkflow 1"

@workflow
def sub_wf1() -> str:
    return task_subwf1()

And for another subworkflow:

from flytekit import task, workflow

@task(container_image="ghcr.io/your_org/subwf2-image:latest")
def task_subwf2() -> str:
    return "Result from subworkflow 2"

@workflow
def sub_wf2() -> str:
    return task_subwf2()

These subworkflows—once registered—can be imported into your orchestration workflow and invoked like ordinary functions. For instance:

from flytekit import workflow
from subwf1 import sub_wf1
from subwf2 import sub_wf2

@workflow
def orchestration_wf() -> None:
    result1 = sub_wf1()
    result2 = sub_wf2()
    # Further processing or triggering subsequent workflows

This approach meets your requirements because: • Each subworkflow is developed, built, and registered independently (with its own container image). • They can later be imported into an orchestration workflow to form an integrated execution plan—with a single point of monitoring for progress and task-level details. Would you like more examples on structuring your project or further details on configuration options? --- Sources: - Registering workflows – Flyte documentation - Multiple images in a workflow – Flyte documentation Sources: - Registering workflows – Flyte documentation - Multiple images in a workflow – Flyte documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

How do you register the orchestration_wf so that it's able to import the subworkflows? @ancient-wolf-19325

It seems likely that the key is to perform a multi‑package registration that includes both the orchestration workflow’s package and the packages for the subworkflows. In your orchestration workflow (for example, in orchestration_wf.py), you would import the subworkflow definitions from their respective packages (e.g., “subwf1” and “subwf2”). Then, when you run the registration command, you include all three packages so that the serialized artifact (used for remote execution) contains the entire dependency graph. For example, suppose your repository is organized like this:

my_project/
  orchestration/
    orchestration_wf.py    # Imports: from subwf1 import sub_wf1, from subwf2 import sub_wf2
  subwf1/
    sub_wf1.py             # Defines sub_wf1 with its own container image
  subwf2/
    sub_wf2.py             # Defines sub_wf2 with its own container image

During registration, you’d run a command similar to:

pyflyte register --non-fast --version t1 \
    --pkgs my_project/orchestration \
    --pkgs my_project/subwf1 \
    --pkgs my_project/subwf2

This command tells Flyte to serialize and register all the workflows found in those packages so that your orchestration workflow can properly import and call the subworkflow definitions. Each subworkflow can still have its own container image specified via the

container_image

parameter in its

@task

decorators, meeting your requirement for different images per workflow. Would you like further details on how to structure your repository or on additional configuration options (such as specifying the source root) to ensure the subworkflow modules get included during registration? --- Sources: - Registering workflows – Flyte documentation - Multiple images in a workflow – Flyte documentation Sources: - Registering workflows – Flyte documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@strong-quill-48244 are you folks using reference workflows or launchplans? Not subworkflows ? They are different in presentation in the ui

Currently we are launching individual workflows using

FlyteRemote

. Which works just fine but lacks in representation in the UI. Trying to figure out if we can use subworksflows as a part of the main workflow but run the subworkflows with dedicated images.