What is the recommended way of structuring your source code if the workflow should reference Python files located directories relative the root path of the repository? I.e., let’s say

workflow.py

needs to access both data and train, do you still recommend this structure and reference everything in

workflow.py

relative root? Probably not explaining the idea that well, but interested in what you think is the best practice with respect to workflows and relative imports/access to other folders containing data/code important to your project.

├── Dockerfile
├── data
│   └── some_folder_with_data
├── train
│   ├── train.py
├── docker_build_and_tag.sh
├── flyte
│   ├── __init__.py
│   └── workflows
│       ├── __init__.py
│       └── workflow.py
├── flyte.config
└── requirements.txt

hi Hampus! your use case makes sense. I don’t think we’ve converged on “best practices” for project structure yet, but if your requirement is that some domain-specific source files live in a directory relative to the

flyte

source, you have a few options: • structure the project as you have it, and make sure that your

PYTHONPATH

env var points to your root project directory in whichever docker image you build so that it has access to the

data

and

train

modules. • combine

data

and

train

into a package (e.g.

my_package

), so you can pip install it locally and on the docker image.

├── Dockerfile
├── <my_package>
│   ├── data
│       └── some_folder_with_data
│   ├── train
│       ├── train.py
├── setup.py  # for my_package
├── docker_build_and_tag.sh
├── flyte
│   ├── __init__.py
│   └── workflows
│       ├── __init__.py
│       └── workflow.py
├── flyte.config
└── requirements.txt

Then you could install and import

my_package

from your tasks/workflows modules

Hi Niels, thank you so much for the input. 🙂 I will do some experimentation and get back if I come across some interesting findings

Awesome! let me know if you have any other questions