I'm working through our monorepo conversion right now and I'm having problems registering flyte pipelines that include cross silo dependencies. For a minimal example, my monorepo has the following structure

pipelines
- src
- - tasks
- - - mytasks.py
- - workflows
- - - myworkflow.py
- - utils
- - - pipeline_utils.py
- shared
- - src
- - - some_utils.py

mytasks.py

has the import

from shared.src.some_utils import util

which runs fine on local but when I package up using

pyflyte register pipelines/src/workflows/

from the root of the monorepo, the packed tarball only includes

pipeline/src/tasks

,

pipeline/src/utils/

and

pipeline/src/workflows

but not

shared

despite references so I get immediate crashes. Has anyone else seen/solved this problem?

I don't necessarily have a solution, but I wonder if the "production flow", as described here, would help? When you call

pyflyte package

you'd provide a container image with all the dependencies, including

shared/src/come_utils.py