seunggs
04/03/2023, 3:14 AMpyflyte package
- pyflyte run
seems to work fine when I add the project root path to sys.path
Samhita Alla
sys.path
. Can you add an __init__.py
file to wf
directory and run your pyflyte package command? Let me know if you're seeing any error.
Your PYTHONPATH has to be the root directory where you have tasks and workflows.seunggs
04/03/2023, 2:06 PM__init__.py
but I’m not able to run pyflyte package
where wf and tasks are. As you can see in the project structure, I had to run it two levels up from where the wf code is because I couldn’t get it to run otherwise. What would be --pkgs
value be if I run it in the same dir as the wf code? I tried using .
for current dir, but that doesn’t work.Samhita Alla
seunggs
04/03/2023, 5:19 PMpyflyte --pkgs wf,main,src package --source project/wf_22_142
/project
/wf_22_142
/src
helpers.py
main.py # where I do `from src.helpers import some_fn`
wf.py # where @workflow is where I do `from main import task_fn`
pyflyte
is executed from parent of the /project
dirModuleNotFoundError: No module named 'src'
--source
option should set the given dir as the project root for importswf.py
where I from main import task_fn
and it seems to work without an issue. Only main.py
importing from src
failswf.py
imports a task from task.py
which in turn imports some functions from a subdirectory src/helpers.py
(e.g. in task.py
, from src.helpers import some_fn
)? @Samhita AllaSamhita Alla
def sum(a, b):
return a + b
main.py
from .src.helpers import sum
from flytekit import task
@task
def main_task(a: int, b: int) -> int:
return sum(a, b)
wf.py
from flytekit import workflow
from .main import main_task
@workflow
def wf(a: int = 10, b: int = 9):
return main_task(a=a, b=b)
Dockerfile
FROM <http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>
# Copy the actual code
COPY wf /root/project/wf
Ran these two commands in `project`'s parent directory:
pyflyte --pkgs project package --image <http://ghcr.io/samhita-alla/flyte-dir-structure:0.0.1|ghcr.io/samhita-alla/flyte-dir-structure:0.0.1> -f
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
seunggs
04/04/2023, 2:44 PMImportError: attempted relative import with no known parent package
) - is there any way to not use relative imports? i.e. from main import main_task
and from src.helpers import sum
rather than from .main…
and from .src…
?/project/wf
dir the package root for python imports, for example?Loading packages ['project'] under source root /source
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'src'
No module named 'src'
where /source
is your /root
)Samhita Alla
seunggs
04/04/2023, 4:08 PMsidetrek/base-flyte
image this Dockerfile is based on just has flytectl installed and flyte-config file):
# Build stage
FROM sidetrek/base-flyte:latest as build
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
# Make sure to use venv
ENV PATH="$VENV/bin:$PATH"
COPY ./project/requirements.txt /root
# Add --no-cache-dir to prevent OOMKilled
RUN pip install --no-cache-dir -r /root/requirements.txt
# Production stage
FROM sidetrek/base-flyte:latest
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
# Add flytectl to PATH
ENV PATH="/bin/flytectl:$PATH"
# Make sure to use venv
ENV PATH="$VENV/bin:$PATH"
# Copy dependencies from build stage
COPY --from=build /opt/venv /opt/venv
# Copy the actual code (again, user's project code)
COPY . /root
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
Samhita Alla
Eduardo Apolinario (eapolinario)
04/04/2023, 4:24 PM___init___.py
file in src
?seunggs
04/04/2023, 4:24 PM__init__.py
in /src
/src
__init__.py
helpers.py
main.py # `from src.helpers import some_fn`
wf_x.py # `from main import task1`
/project/wf
to accomodate flyte packaging/registration:
/project
/wf
/src
__init__.py
helpers.py
main.py
wf_x.py
__init__.py
Dockerfile
requirements.txt
And we’re currently running pyflyte --pkgs wf_x package --source project/wf
from parent dir of the /project
, which works fine except during execution, we get no module named src
errorSamhita Alla
wf_x
but you need to access src
while executing your code. Try packaging wf
.seunggs
04/04/2023, 5:02 PMproject/wf
, this errors out saying there’s no module named wf
(probably because you’re already in that dir)project
, and I get No module named 'src'
errorSamhita Alla
--source
, and can you run this command in `project`'s parent directory?
pyflyte --pkgs project package --image <your-image> -f
The __init__.py
file needs to be present in wf
, not project
. Ensure the Dockerfile is copying the code in wf
to /root/project/wf
.seunggs
04/05/2023, 3:01 PMLoading packages ['project'] under source root /workspace/source
No module named 'src'
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'src'
/workspace/source
is the root folder inside CI/CD where /project
is.
Confirmed that the structure is this inside the CI/CD:
/project
/wf
/src
__init__.py
helpers.py
__init__.py
main.py
wf.py
And ran pyflyte --pkgs project package …
without --source
from parent of /project
.
Using relative import - i.e. in main.py
, from .src.helpers import some_fn
Samhita Alla
seunggs
04/05/2023, 4:19 PMproject
- shouldn’t that package all the subdirectories and files?src
directly as a package via --pkgs
option too but that doesn’t seem to work either. It only seems to package .py
files?--pkgs
?Samhita Alla
Also, I’d much prefer to not use relative import if it’s at all possible - I think it’s strongly preferred to have a parity between dev and production and the user shouldn’t have to adhere to a specific project dir structure to accommodate flyte deployment (or be coerced to use relative imports when locally, they can run the code without them).
seunggs
04/05/2023, 5:25 PMattempted relative import with no known parent package
pyflyte package
and pyflyte run
- No module named 'src'
. Running from parent of the /project
in the above dir structureSamhita Alla
seunggs
04/07/2023, 2:53 AMattempted relative import with no known parent package
error, I wasn’t able to get relative import working. For various reasons, I don’t think we can use the relative imports for our needs.
The only way I could get this to work was to do something like from <http://project.wf|project.wf>.src.helpers import some_fn
when I run my project in the parent dir of project
(with no --source and __init__.py
in /project
and in /wf
). This is an acceptable workaround for now, but I’d much rather be able to set /project/wf
dir as project root for python import purposes.
So I’d love to get on a call if you can help me find a way to do this, but otherwise, I am not sure I should waste any more of your time.
Let me know if you think it’s still worth it to get on a call. Either way, thank you so much for your help regarding this problem. It’s much appreciated!!Samhita Alla
The only way I could get this to work was to do something likeGood to know that this is working for you.when I run my project in the parent dir offrom <http://project.wf|project.wf>.src.helpers import some_fn
(with no --source andproject
in__init__.py
and in/project
)./wf
This is an acceptable workaround for now, but I’d much rather be able to setYou can setdir as project root for python import purposes./project/wf
wf
as the project root if all your workflows and dependent tasks and libraries are present in the same folder. If not, this won't work.
e.g.
main.py
from wf.src.helpers import sum
from flytekit import task
@task
def main_task(a: int, b: int) -> int:
return sum(a, b)
wf.py
from flytekit import workflow
from wf.main import main_task
@workflow
def wf(a: int = 10, b: int = 9):
return main_task(a=a, b=b)
I'm able to successfully package code when I run pyflyte package
command in the project
directory.
Also, I agree that relative import isn't a feasible import technique. You should either import from the project root or do relative imports. Let me know if this still isn't clear to you and we can hop on a call!seunggs
04/07/2023, 2:32 PM/wf
as project root if there are subdirectories?
This forces a user to either have a specific dir structure (i.e. /project/wf
), which requires a prefix for all imports (i.e. from <http://project.wf|project.wf>.main…
) or if you make /wf
project root, you have to not have any subdirectories, which prevents the user from having any modularized code. Either way, flyte loses parity with a regular python code.Samhita Alla
wf
.seunggs
04/07/2023, 2:36 PMwf
as the project root if all your workflows and dependent tasks and libraries are present in the same folder. If not, this won’t work.”Samhita Alla
wf
and that should work!seunggs
04/07/2023, 2:37 PMwf
- but I tried this and it doesn’t workSamhita Alla
seunggs
04/07/2023, 2:37 PM/project/wf
as a --src so it can be the project root, then what would --pkgs value be?/project
/wf
/src
__init__.py
helpers.py
__init__.py
main.py
wf.py
--pkgs wf
as well as --pkgs wf, main, src
and neither worksSamhita Alla
project
not project/wf
'cause you need to package the whole wf
directoryseunggs
04/07/2023, 2:38 PMfrom main …
doesn’t workfrom wf.main ...
Samhita Alla
seunggs
04/07/2023, 2:39 PMfrom main …
Samhita Alla
/project
/wf
/src
__init__.py
helpers.py
__init__.py
main.py
wf.py
can we package code at the wf
level by not importing code from other modules relative to wf
? For example, if we want to import code in wf.py
from main.py
, we need to include from wf.main ...
but not from main ...
'cause the latter results in no module error.