https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
s

seunggs

03/16/2023, 5:33 PM
Hi team, I can’t for the life of me get the 
pyflyte package
to work without causing module import issues - any help would be greatly appreciated. The project is in 
/project
(root dir) and inside is 
/wf
dir with 
wf_10.py
where the workflow code lives, where I import tasks from 
main.py
in the same dir. I’m running pyflyte from 
/project
dir with this command 
pyflyte --pkgs wf packages ...
and the packaging fails with:
Copy code
Loading packages ['wf'] under source root /project
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'main'
No module named 'main'
k

Kevin Su

03/16/2023, 6:26 PM
is there an empty 
__init__.py
in 
/project
?
s

seunggs

03/16/2023, 6:26 PM
No nothing in 
project
Is that required?
k

Kevin Su

03/16/2023, 6:27 PM
yes, IIRC
could you give it a try
s

seunggs

03/16/2023, 6:28 PM
Sure - one more question. What if I put the files in 
/project
directly and run 
pyflyte package
from there? What should I put as 
--pkgs
option value?
k

Kevin Su

03/16/2023, 6:32 PM
--pkgs /project/wf
s

seunggs

03/16/2023, 6:33 PM
The docs say I should use dot notation - is that no longer the case?
Also tried adding 
__init__.py
to 
/project
folder but same error:
Copy code
Loading packages ['wf'] under source root /project
No module named 'main'
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'main'
Does the 
--pkgs
expect absolute path to the package? I assume you can specify relative path from cwd no?
k

Kevin Su

03/16/2023, 6:43 PM
yes, sorry. should be relative. let me try on my end.
s

seunggs

03/16/2023, 6:44 PM
No worries - yeah I tried running 
--pkgs .
for running 
wf_10.py
from project root (where I ran 
pyflyte
command), but that doesn’t work. 
--pkgs ''
doesn’t work either
Adding 
__init__.py
in the 
/project
and then running 
pyflyte --pkgs wf package ...
there with the code inside 
/wf
didn’t work either
k

Kevin Su

03/16/2023, 6:56 PM
pyflyte --config ~/.flyte/config-remote.yaml --pkgs project.wf package --image ghcr.io/flyteorg/flytekit:py3.9-latest
└── project ├── init.py ├── pycache │ └── init.cpython-39.pyc └── wf ├── init.py ├── pycache │ ├── init.cpython-39.pyc │ └── test.cpython-39.pyc └── test.py
I ran it in the parent folder of project
s

seunggs

03/16/2023, 7:09 PM
Oh you can’t run it in the project folder?
i.e. 
pyflyte --pkgs wf package …
?
Also I think the test might not be quite right since I’m importing tasks from 
main.py
in my workflow file 
wf.py
- and the error is 
main
module does not exist
So I’m assuming 
from main import task1
in 
wf.py
is not working because 
pyflyte
was run in 
/project
- so I think it’s expecting my 
from main
to be 
from wf.main
, but I’d like to just use 
main
as the module
Hope that makes sense?
So to summarize import inside 
wf.py
is the problem I’m having
I thought 
--pkgs
sets the path in PYTHON_PATH, but it doesn’t seem to?
k

Kevin Su

03/16/2023, 8:40 PM
could you try to add 
--source ./project/wf
in the command,
you’re using fast packing, right?
s

seunggs

03/16/2023, 8:58 PM
No not using fast packing - building this inside CI/CD for initial packaging/deployment
I’ll try the --source and get back to you
So inside 
/project
, I tried 
pyflyte --pkgs wf package --source ./wf
and using it now gives an error 
No module named 'wf'
instead of 
No module named 'main'
- so it looks like 
--source
sets the project folder, but by doing that, 
--pkgs
is now breaking
I’m just not sure what to set 
--pkgs
in this case though since neither the current dir 
.
or empty 
''
works
@Kevin Su
@Kevin Su I got it to work but at least as far as I can tell, you can’t have wf.py and main.py (where task functions are) in the same dir - I got it to work after putting the wf.py in 
/project/wf
whereas main.py is in 
/project
. That way I could run 
pyflyte
from 
/project
and then set 
--pkgs
to 
wf
. This seems like a bug to me. I should be able to specify the cwd as -
-pkgs
(e.g. 
--pkgs .
)
Powered by