sticky-angle-28419
04/03/2023, 3:14 AMpyflyte package
- pyflyte run
seems to work fine when I add the project root path to sys.path
tall-lock-23197
sys.path
. Can you add an __init__.py
file to wf
directory and run your pyflyte package command? Let me know if you're seeing any error.
Your PYTHONPATH has to be the root directory where you have tasks and workflows.sticky-angle-28419
04/03/2023, 2:06 PM__init__.py
but I’m not able to run pyflyte package
where wf and tasks are. As you can see in the project structure, I had to run it two levels up from where the wf code is because I couldn’t get it to run otherwise. What would be --pkgs
value be if I run it in the same dir as the wf code? I tried using .
for current dir, but that doesn’t work.tall-lock-23197
sticky-angle-28419
04/03/2023, 5:19 PMsticky-angle-28419
04/03/2023, 5:21 PMpyflyte --pkgs wf,main,src package --source project/wf_22_142
sticky-angle-28419
04/03/2023, 5:23 PM/project
/wf_22_142
/src
helpers.py
main.py # where I do `from src.helpers import some_fn`
wf.py # where @workflow is where I do `from main import task_fn`
sticky-angle-28419
04/03/2023, 5:23 PMpyflyte
is executed from parent of the /project
dirsticky-angle-28419
04/03/2023, 5:30 PMModuleNotFoundError: No module named 'src'
sticky-angle-28419
04/03/2023, 5:42 PM--source
option should set the given dir as the project root for importssticky-angle-28419
04/03/2023, 5:51 PMwf.py
where I from main import task_fn
and it seems to work without an issue. Only main.py
importing from src
failssticky-angle-28419
04/03/2023, 8:11 PMsticky-angle-28419
04/03/2023, 8:11 PMsticky-angle-28419
04/03/2023, 8:12 PMsticky-angle-28419
04/04/2023, 2:59 AMwf.py
imports a task from task.py
which in turn imports some functions from a subdirectory src/helpers.py
(e.g. in task.py
, from src.helpers import some_fn
)? @tall-lock-23197sticky-angle-28419
04/04/2023, 3:01 AMsticky-angle-28419
04/04/2023, 3:02 AMtall-lock-23197
def sum(a, b):
return a + b
main.py
from .src.helpers import sum
from flytekit import task
@task
def main_task(a: int, b: int) -> int:
return sum(a, b)
wf.py
from flytekit import workflow
from .main import main_task
@workflow
def wf(a: int = 10, b: int = 9):
return main_task(a=a, b=b)
Dockerfile
FROM <http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>
# Copy the actual code
COPY wf /root/project/wf
Ran these two commands in `project`'s parent directory:
pyflyte --pkgs project package --image <http://ghcr.io/samhita-alla/flyte-dir-structure:0.0.1|ghcr.io/samhita-alla/flyte-dir-structure:0.0.1> -f
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
sticky-angle-28419
04/04/2023, 2:44 PMImportError: attempted relative import with no known parent package
) - is there any way to not use relative imports? i.e. from main import main_task
and from src.helpers import sum
rather than from .main…
and from .src…
?sticky-angle-28419
04/04/2023, 2:46 PMsticky-angle-28419
04/04/2023, 2:48 PM/project/wf
dir the package root for python imports, for example?sticky-angle-28419
04/04/2023, 2:49 PMsticky-angle-28419
04/04/2023, 3:19 PMLoading packages ['project'] under source root /source
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'src'
No module named 'src'
where /source
is your /root
)tall-lock-23197
sticky-angle-28419
04/04/2023, 4:08 PMsidetrek/base-flyte
image this Dockerfile is based on just has flytectl installed and flyte-config file):
# Build stage
FROM sidetrek/base-flyte:latest as build
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
# Make sure to use venv
ENV PATH="$VENV/bin:$PATH"
COPY ./project/requirements.txt /root
# Add --no-cache-dir to prevent OOMKilled
RUN pip install --no-cache-dir -r /root/requirements.txt
# Production stage
FROM sidetrek/base-flyte:latest
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
# Add flytectl to PATH
ENV PATH="/bin/flytectl:$PATH"
# Make sure to use venv
ENV PATH="$VENV/bin:$PATH"
# Copy dependencies from build stage
COPY --from=build /opt/venv /opt/venv
# Copy the actual code (again, user's project code)
COPY . /root
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
sticky-angle-28419
04/04/2023, 4:08 PMtall-lock-23197
tall-lock-23197
high-accountant-32689
04/04/2023, 4:24 PM___init___.py
file in src
?sticky-angle-28419
04/04/2023, 4:24 PM__init__.py
in /src
sticky-angle-28419
04/04/2023, 4:26 PMsticky-angle-28419
04/04/2023, 4:27 PM/src
__init__.py
helpers.py
main.py # `from src.helpers import some_fn`
wf_x.py # `from main import task1`
sticky-angle-28419
04/04/2023, 4:29 PM/project/wf
to accomodate flyte packaging/registration:
/project
/wf
/src
__init__.py
helpers.py
main.py
wf_x.py
__init__.py
Dockerfile
requirements.txt
And we’re currently running pyflyte --pkgs wf_x package --source project/wf
from parent dir of the /project
, which works fine except during execution, we get no module named src
errorsticky-angle-28419
04/04/2023, 4:33 PMtall-lock-23197
wf_x
but you need to access src
while executing your code. Try packaging wf
.sticky-angle-28419
04/04/2023, 5:02 PMproject/wf
, this errors out saying there’s no module named wf
(probably because you’re already in that dir)sticky-angle-28419
04/04/2023, 5:16 PMproject
, and I get No module named 'src'
errortall-lock-23197
--source
, and can you run this command in `project`'s parent directory?
pyflyte --pkgs project package --image <your-image> -f
The __init__.py
file needs to be present in wf
, not project
. Ensure the Dockerfile is copying the code in wf
to /root/project/wf
.sticky-angle-28419
04/05/2023, 3:01 PMLoading packages ['project'] under source root /workspace/source
No module named 'src'
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'src'
/workspace/source
is the root folder inside CI/CD where /project
is.
Confirmed that the structure is this inside the CI/CD:
/project
/wf
/src
__init__.py
helpers.py
__init__.py
main.py
wf.py
And ran pyflyte --pkgs project package …
without --source
from parent of /project
.
Using relative import - i.e. in main.py
, from .src.helpers import some_fn
sticky-angle-28419
04/05/2023, 4:15 PMsticky-angle-28419
04/05/2023, 4:15 PMtall-lock-23197
sticky-angle-28419
04/05/2023, 4:19 PMproject
- shouldn’t that package all the subdirectories and files?sticky-angle-28419
04/05/2023, 4:20 PMsticky-angle-28419
04/05/2023, 4:20 PMsrc
directly as a package via --pkgs
option too but that doesn’t seem to work either. It only seems to package .py
files?sticky-angle-28419
04/05/2023, 4:21 PM--pkgs
?tall-lock-23197
tall-lock-23197
Also, I’d much prefer to not use relative import if it’s at all possible - I think it’s strongly preferred to have a parity between dev and production and the user shouldn’t have to adhere to a specific project dir structure to accommodate flyte deployment (or be coerced to use relative imports when locally, they can run the code without them).
sticky-angle-28419
04/05/2023, 5:25 PMattempted relative import with no known parent package
sticky-angle-28419
04/05/2023, 5:30 PMsticky-angle-28419
04/05/2023, 7:49 PMpyflyte package
and pyflyte run
- No module named 'src'
. Running from parent of the /project
in the above dir structuresticky-angle-28419
04/05/2023, 7:58 PMtall-lock-23197
tall-lock-23197
sticky-angle-28419
04/07/2023, 2:53 AMattempted relative import with no known parent package
error, I wasn’t able to get relative import working. For various reasons, I don’t think we can use the relative imports for our needs.
The only way I could get this to work was to do something like from <http://project.wf|project.wf>.src.helpers import some_fn
when I run my project in the parent dir of project
(with no --source and __init__.py
in /project
and in /wf
). This is an acceptable workaround for now, but I’d much rather be able to set /project/wf
dir as project root for python import purposes.
So I’d love to get on a call if you can help me find a way to do this, but otherwise, I am not sure I should waste any more of your time.
Let me know if you think it’s still worth it to get on a call. Either way, thank you so much for your help regarding this problem. It’s much appreciated!!tall-lock-23197
The only way I could get this to work was to do something likeGood to know that this is working for you.when I run my project in the parent dir offrom <http://project.wf|project.wf>.src.helpers import some_fn
(with no --source andproject
in__init__.py
and in/project
)./wf
This is an acceptable workaround for now, but I’d much rather be able to setYou can setdir as project root for python import purposes./project/wf
wf
as the project root if all your workflows and dependent tasks and libraries are present in the same folder. If not, this won't work.
e.g.
main.py
from wf.src.helpers import sum
from flytekit import task
@task
def main_task(a: int, b: int) -> int:
return sum(a, b)
wf.py
from flytekit import workflow
from wf.main import main_task
@workflow
def wf(a: int = 10, b: int = 9):
return main_task(a=a, b=b)
I'm able to successfully package code when I run pyflyte package
command in the project
directory.
Also, I agree that relative import isn't a feasible import technique. You should either import from the project root or do relative imports. Let me know if this still isn't clear to you and we can hop on a call!sticky-angle-28419
04/07/2023, 2:32 PM/wf
as project root if there are subdirectories?
This forces a user to either have a specific dir structure (i.e. /project/wf
), which requires a prefix for all imports (i.e. from <http://project.wf|project.wf>.main…
) or if you make /wf
project root, you have to not have any subdirectories, which prevents the user from having any modularized code. Either way, flyte loses parity with a regular python code.tall-lock-23197
wf
.sticky-angle-28419
04/07/2023, 2:36 PMwf
as the project root if all your workflows and dependent tasks and libraries are present in the same folder. If not, this won’t work.”tall-lock-23197
wf
and that should work!sticky-angle-28419
04/07/2023, 2:37 PMwf
- but I tried this and it doesn’t worktall-lock-23197
sticky-angle-28419
04/07/2023, 2:37 PM/project/wf
as a --src so it can be the project root, then what would --pkgs value be?sticky-angle-28419
04/07/2023, 2:37 PM/project
/wf
/src
__init__.py
helpers.py
__init__.py
main.py
wf.py
sticky-angle-28419
04/07/2023, 2:38 PM--pkgs wf
as well as --pkgs wf, main, src
and neither workstall-lock-23197
project
not project/wf
'cause you need to package the whole wf
directorysticky-angle-28419
04/07/2023, 2:38 PMfrom main …
doesn’t worksticky-angle-28419
04/07/2023, 2:38 PMfrom wf.main ...
tall-lock-23197
sticky-angle-28419
04/07/2023, 2:39 PMsticky-angle-28419
04/07/2023, 2:39 PMfrom main …
sticky-angle-28419
04/07/2023, 2:39 PMtall-lock-23197
tall-lock-23197
/project
/wf
/src
__init__.py
helpers.py
__init__.py
main.py
wf.py
can we package code at the wf
level by not importing code from other modules relative to wf
? For example, if we want to import code in wf.py
from main.py
, we need to include from wf.main ...
but not from main ...
'cause the latter results in no module error.