how do folks handle multiple sets of conflicting dependencies in a monorepo of flyte workflows and tasks? I want to have some shared set of reusable building blocks that relies on a shared set of dependencies (let’s say this just relies on boto3 and nothing else) and then give some way to allow a given workflow to have its own set of additional dependencies (let’s say a specific version of numpy). other workflows may conflict and want a different version of numpy when fast registration occurs, I want both the code of the shared reusable building blocks and the target workflow to end up in s3

@curved-whale-1505 you can separate them into multiple registrations. and then use

reference tasks

@curved-whale-1505, to expand on that, there are a few different ways: 1. You can use separate ImageSpecs with different dependencies and annotate the task with that

@task(container_image=my_image_1)

and the system will automatically use the right image with the right dependencies. If you do this, you don't have to create separate workflows as Flyte allows you to mix images in the same workflow 2. You can use different workflows/tasks and reference them using Reference Tasks/Workflows as Ketan pointed out 3. We have recently added a feature called

with_runtime_packages

(Docs will be part of the upcoming official flyte release) that allows you to use

uv run

to install packages on the fly before a task run

@task(container_image=img.with_runtime_packages(["numpy==..."]))
def my_task()...

Note option #3 does incur runtime penalty as it'll install the packages on the fly every time this task runs.

So I was hoping to have each ImageSpec point at a local

uv.lock

file and then ensure I can always run

pyflyte run ...

without

--remote

to replicate the entire workflow locally. Let’s say I have the following structure (pseudocode):

monorepo/
├── common_utils
│   ├── pyproject.toml
│   └── uv.lock
├── project_a
│   ├── pyproject.toml
│   └── uv.lock
├── project_b
│   ├── pyproject.toml
│   └── uv.lock

In this example, the dependencies of

common_utils

is a strict subset of

project_a

and

project_b

. And

project_a

and

project_b

have conflicting dependencies (lets stick with conflicting numpy as per the example before). Now ideally I want want: 1) when I run

project_a

, the code for both

common_utils

and

project_a

to be fast registered into s3 2) the ability to run

pyflyte run ...

without

--remote

to replicate the entire workflow(s) locally within

project_a

and

project_b

@curved-whale-1505 While I haven't done this yet (maybe in the next couple months), I'm planning to write some plugins for a monorepo build system (for me Pants) to help with this kind of stuff.

aha.. you are right, locally the assumption is you can have a single venv with the right packages installed... Is there an assumption in your example that there is a dependency between project_a and project_b (hence the conflicts)? Is it possible to structure things so that a workflow can independently run (can be unit tested) while having a separate project_integration that ties things together using any of the above methods... this way only that workflow won't run locally It's an interesting problem though, we are rethinking some of the execution engine fundamentals and may have a better solution cooking 🍳 (stay tuned 🙂)