ancient-wolf-19325
08/16/2024, 10:06 AM>># QuestionHi! I have a project directory with the following structure:
├── conf
├── data
└── src
When I register my workflow (I want to use fast registration), I need files in conf and data TO BE PRESENT during execution. The workflow itself is defined under src/my_package/workflows/workflow.py . At execution time in Flyte it seems like only files from src are present in the container. How to include additional files outside of the src folder? They change too during interations on the workflow. I expected that's a basic scenario that is supported to enable fast development cycle (as stated here https://docs.flyte.org/en/latest/flyte_fundamentals/registering_workflows.html#fast-registration )
# Answer
To include additional files outside of the src folder during fast registration, you can use the --include flag with pyflyte run. This flag allows you to specify additional directories or files to be included in the tarball that is fast-registered and made available during execution.
Here's how you can modify your command to include the conf and data directories:
pyflyte run --remote --include conf --include data src/my_package/workflows/workflow.py
This command will ensure that the conf and data directories are included in the tarball and available during execution.
Would you like more details on this workaround or any other aspect of Flyte?
Sources:
- Flyte GitHub Issue - UX improvement: add --copy flag to pyflyte run for more targeted file copying during fast registration
- Flyte Documentation - Registering Workflows
Slack Link
https://flyte-org.slack.com/archives/CP2HDHKE1/p1723802756.623569 👍 0 button 👎 1 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.