Hi again :wave: We've been going around in circle...
# flyte-support
a
Hi again 👋 We've been going around in circles with this one, and I was curious if anyone has any tips. The question is around ImageSpecs, local modules, pyflyte package and flytectl register - I'll add more details to the thread 🧵
We've had a lot of problems getting our project detected as a module during registration/packaging, and have tried a few configurations now that were very slow to build. We've just discovered the 'default' ImageSpec builder that was added to flyte recently, which has sped up our builds massively 🚀 But when we try to import the main repository module into our task code (following packaging), we hit the dreaded "ModuleNotFoundError: no module named 'mycoolproject'". Our folder structure is as below, and the commands we use for packaging are:
Copy code
pyflyte --pkgs mycoolproject package -f
flytectl register files --archive flyte-package.tgz --version custom-version
The tasks work fine during
pyflyte run --remote
, but fail with the ModuleNotFound error after the production packaging process. We can get around the ModuleNotFoundError by building a custom builder that copies the project directory to the image - but ideally we'd like to use the default builder. Is there an extra command or step we're missing to ensure that flyte tasks correctly have access to the surrounding module? An example project structure is below....
Copy code
/my-cool-project
|_ src
   |_coolproject
       |_ flyte
              image_specs.py
       |_ tasks
              tasks_1.py
              tasks_2.py
       |_ utils
              common.py
       |_ workflows
              workflows.py
       constants.py

tasks_1.py
from coolproject.constants import USEFUL_CONSTANT
from coolproject.flyte.image_spec import CUSTOM_IMAGE
from coolproject.utils.common import handy_helper

@task
def task_1(container_image=CUSTOM_IMAGE) -> None:
    handy_helper(USEFUL_CONSTANT)
task_1 would fail with the 'ModuleNotFound' error following the packaging steps...
a
@abundant-judge-84756 have you tried including
--copy-all
in the
pyflyte
command?
g
“ModuleNotFoundError: no module named ‘mycoolproject’“. Our folder structure is as below, and the commands we use for packaging are:
is your code in the image built by default image builder?
a
@average-finland-92144 Unfortunately
--copy-all
works if we're just running one task using
pyflyte run
, but not if we want to package the whole repository using
pyflyte package
(we get a 'no such option' error). @glamorous-carpet-83516 We're using the builder labelled 'default' that was added here., ie.
Copy code
CUSTOM_IMAGE = ImageSpec(
    builder="default"
)
g
i mean if you download the image, and go into the container (docker run --it -rm … bash). did you see your module?
a
@glamorous-carpet-83516 Good suggestion - I did some comparisons of the images built by
pyflyte run
and
pyflyte register
and tried some new command variants - I think I might have now gotten the pyflyte package to work by combining --pkgs and --source! So far looking good with this version of the command 🤞
Copy code
pyflyte --pkgs mycoolproject package --source ./src -f
🎉 1
g
cool. good to know that. 👍
🙏 1
a
Thanks again for the help!
f
We need to probably document this correctly
💯 1