@ancient-wolf-19325 how can I load custom packages from subfolders in a flyte run?

To load custom packages from subfolders in a Flyte run, you can use the

--copy-all

flag with the

pyflyte run

command. This flag ensures that all the source files, including those in subfolders, are present in the container running the workflow.

pyflyte run --remote --copy-all <workflow_file> <workflow_function> --<workflow_args>

For more targeted inclusion of files, you can use the

--include

flag to specify directories or files to be included during fast registration. This is useful for data files or compiled files.

pyflyte run --remote --include <dir1> --include <dir2> <workflow_file> <workflow_function> --<workflow_args>

Would you like more detail on either of these methods? Sources: - Flyte GitHub Issue - UX improvement: add --copy flag to pyflyte run - Flyte GitHub Issue - UX improvement: pyflyte run includes imported local modules 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 still fails when I use importlib.import_module

Using

importlib.import_module

can sometimes fail due to the way Python handles module imports, especially in distributed environments like Flyte. This issue can be related to the module not being in the Python path or the module not being packaged correctly. To address this, ensure the following: 1. Correct File Structure: Ensure your project structure is correct and includes

__init__.py

files to make directories recognizable as packages. 2. Include All Necessary Files: Use the

--copy-all

or

--include

flags to ensure all necessary files and directories are included in the container. Example using `--copy-all`:

pyflyte run --remote --copy-all <workflow_file> <workflow_function> --<workflow_args>

Example using `--include`:

pyflyte run --remote --include <dir1> --include <dir2> <workflow_file> <workflow_function> --<workflow_args>

If the issue persists, it might be due to the specific way

importlib.import_module

is being used. Ensure the module path is correct and the module is available in the Python path. Would you like more detailed steps or examples? Sources: - Flyte GitHub Issue - Task fails to import modules in `pyflyte-map-execute` mode - Flyte GitHub Issue - UX improvement: pyflyte run includes imported local modules 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.