https://flyte.org logo
#ask-the-community
Title
# ask-the-community
s

seunggs

03/16/2023, 5:33 PM
Hi team, I can’t for the life of me get the
pyflyte package
to work without causing module import issues - any help would be greatly appreciated. The project is in
/project
(root dir) and inside is
/wf
dir with
wf_10.py
where the workflow code lives, where I import tasks from
main.py
in the same dir. I’m running pyflyte from
/project
dir with this command
pyflyte --pkgs wf packages ...
and the packaging fails with:
Copy code
Loading packages ['wf'] under source root /project
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'main'
No module named 'main'
k

Kevin Su

03/16/2023, 6:26 PM
is there an empty
__init__.py
in
/project
?
s

seunggs

03/16/2023, 6:26 PM
No nothing in
project
Is that required?
k

Kevin Su

03/16/2023, 6:27 PM
yes, IIRC
could you give it a try
s

seunggs

03/16/2023, 6:28 PM
Sure - one more question. What if I put the files in
/project
directly and run
pyflyte package
from there? What should I put as
--pkgs
option value?
k

Kevin Su

03/16/2023, 6:32 PM
--pkgs /project/wf
s

seunggs

03/16/2023, 6:33 PM
The docs say I should use dot notation - is that no longer the case?
Also tried adding
__init__.py
to
/project
folder but same error:
Copy code
Loading packages ['wf'] under source root /project
No module named 'main'
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'main'
Does the
--pkgs
expect absolute path to the package? I assume you can specify relative path from cwd no?
k

Kevin Su

03/16/2023, 6:43 PM
yes, sorry. should be relative. let me try on my end.
s

seunggs

03/16/2023, 6:44 PM
No worries - yeah I tried running
--pkgs .
for running
wf_10.py
from project root (where I ran
pyflyte
command), but that doesn’t work.
--pkgs ''
doesn’t work either
Adding
__init__.py
in the
/project
and then running
pyflyte --pkgs wf package ...
there with the code inside
/wf
didn’t work either
k

Kevin Su

03/16/2023, 6:56 PM
pyflyte --config ~/.flyte/config-remote.yaml --pkgs project.wf package --image ghcr.io/flyteorg/flytekit:py3.9-latest
└── project ├── init.py ├── pycache │ └── init.cpython-39.pyc └── wf ├── init.py ├── pycache │ ├── init.cpython-39.pyc │ └── test.cpython-39.pyc └── test.py
I ran it in the parent folder of project
s

seunggs

03/16/2023, 7:09 PM
Oh you can’t run it in the project folder?
i.e.
pyflyte --pkgs wf package …
?
Also I think the test might not be quite right since I’m importing tasks from
main.py
in my workflow file
wf.py
- and the error is
main
module does not exist
So I’m assuming
from main import task1
in
wf.py
is not working because
pyflyte
was run in
/project
- so I think it’s expecting my
from main
to be
from wf.main
, but I’d like to just use
main
as the module
Hope that makes sense?
So to summarize import inside
wf.py
is the problem I’m having
I thought
--pkgs
sets the path in PYTHON_PATH, but it doesn’t seem to?
k

Kevin Su

03/16/2023, 8:40 PM
could you try to add
--source ./project/wf
in the command,
you’re using fast packing, right?
s

seunggs

03/16/2023, 8:58 PM
No not using fast packing - building this inside CI/CD for initial packaging/deployment
I’ll try the --source and get back to you
So inside
/project
, I tried
pyflyte --pkgs wf package --source ./wf
and using it now gives an error
No module named 'wf'
instead of
No module named 'main'
- so it looks like
--source
sets the project folder, but by doing that,
--pkgs
is now breaking
I’m just not sure what to set
--pkgs
in this case though since neither the current dir
.
or empty
''
works
@Kevin Su
@Kevin Su I got it to work but at least as far as I can tell, you can’t have wf.py and main.py (where task functions are) in the same dir - I got it to work after putting the wf.py in
/project/wf
whereas main.py is in
/project
. That way I could run
pyflyte
from
/project
and then set
--pkgs
to
wf
. This seems like a bug to me. I should be able to specify the cwd as -
-pkgs
(e.g.
--pkgs .
)