What is the recommended way of structuring your so...
# announcements
h
What is the recommended way of structuring your source code if the workflow should reference Python files located directories relative the root path of the repository? I.e., let’s say
workflow.py
needs to access both data and train, do you still recommend this structure and reference everything in
workflow.py
relative root? Probably not explaining the idea that well, but interested in what you think is the best practice with respect to workflows and relative imports/access to other folders containing data/code important to your project.
Copy code
├── Dockerfile
├── data
│   └── some_folder_with_data
├── train
│   ├── train.py
├── docker_build_and_tag.sh
├── flyte
│   ├── __init__.py
│   └── workflows
│       ├── __init__.py
│       └── workflow.py
├── flyte.config
└── requirements.txt
n
hi Hampus! your use case makes sense. I don’t think we’ve converged on “best practices” for project structure yet, but if your requirement is that some domain-specific source files live in a directory relative to the
flyte
source, you have a few options: • structure the project as you have it, and make sure that your
PYTHONPATH
env var points to your root project directory in whichever docker image you build so that it has access to the
data
and
train
modules. • combine
data
and
train
into a package (e.g.
my_package
), so you can pip install it locally and on the docker image.
Copy code
├── Dockerfile
├── <my_package>
│   ├── data
│       └── some_folder_with_data
│   ├── train
│       ├── train.py
├── setup.py  # for my_package
├── docker_build_and_tag.sh
├── flyte
│   ├── __init__.py
│   └── workflows
│       ├── __init__.py
│       └── workflow.py
├── flyte.config
└── requirements.txt
Then you could install and import
my_package
from your tasks/workflows modules
h
Hi Niels, thank you so much for the input. 🙂 I will do some experimentation and get back if I come across some interesting findings
👍 1
n
Awesome! let me know if you have any other questions
259 Views