abundant-judge-84756
08/15/2024, 6:15 PMabundant-judge-84756
08/15/2024, 6:15 PMpyflyte --pkgs mycoolproject package -f
flytectl register files --archive flyte-package.tgz --version custom-version
The tasks work fine during pyflyte run --remote
, but fail with the ModuleNotFound error after the production packaging process.
We can get around the ModuleNotFoundError by building a custom builder that copies the project directory to the image - but ideally we'd like to use the default builder. Is there an extra command or step we're missing to ensure that flyte tasks correctly have access to the surrounding module?
An example project structure is below....abundant-judge-84756
08/15/2024, 6:16 PM/my-cool-project
|_ src
|_coolproject
|_ flyte
image_specs.py
|_ tasks
tasks_1.py
tasks_2.py
|_ utils
common.py
|_ workflows
workflows.py
constants.py
tasks_1.py
from coolproject.constants import USEFUL_CONSTANT
from coolproject.flyte.image_spec import CUSTOM_IMAGE
from coolproject.utils.common import handy_helper
@task
def task_1(container_image=CUSTOM_IMAGE) -> None:
handy_helper(USEFUL_CONSTANT)
task_1 would fail with the 'ModuleNotFound' error following the packaging steps...average-finland-92144
08/15/2024, 6:35 PM--copy-all
in the pyflyte
command?glamorous-carpet-83516
08/15/2024, 9:13 PM“ModuleNotFoundError: no module named ‘mycoolproject’“. Our folder structure is as below, and the commands we use for packaging are:is your code in the image built by default image builder?
abundant-judge-84756
08/16/2024, 7:03 AM--copy-all
works if we're just running one task using pyflyte run
, but not if we want to package the whole repository using pyflyte package
(we get a 'no such option' error).
@glamorous-carpet-83516 We're using the builder labelled 'default' that was added here., ie.
CUSTOM_IMAGE = ImageSpec(
builder="default"
)
glamorous-carpet-83516
08/16/2024, 8:21 AMabundant-judge-84756
08/16/2024, 8:41 AMpyflyte run
and pyflyte register
and tried some new command variants - I think I might have now gotten the pyflyte package to work by combining --pkgs and --source! So far looking good with this version of the command 🤞
pyflyte --pkgs mycoolproject package --source ./src -f
glamorous-carpet-83516
08/16/2024, 8:44 AMabundant-judge-84756
08/16/2024, 8:49 AMfreezing-airport-6809