https://flyte.org logo
f

Frank Shen

10/06/2022, 6:36 PM
I am not able to find flyte documentation for the resolution of this issue. Could anyone share the root cause of the error and the fix?
n

Niels Bantilan

10/06/2022, 7:06 PM
hi @Frank Shen can you share what you’re running to get the error? As @Kevin Su mentioned you can use the pyflyte register command to package and register your workflow to your Flyte cluster
Copy code
pyflyte register --image <image>:<version> <workflows_directory>
Where
<workflows_directory>
is the root directory containing all of the code you want to ship with your image
f

Frank Shen

10/06/2022, 7:14 PM
Sure, @Niels Bantilan The command is:
Copy code
pyflyte run --remote refer_subworkflow_in_lib.py parent_wf --a 9
In refer_subworkflow_in_lib.py file:
Copy code
from lib.subwf import op, t1, my_subwf
I run command (last param flyte is the folder name):
Copy code
pyflyte register --image dai/mlp/flyte/demo_image:1 flyte
And I got:
Copy code
from lib.subwf import op, t1, my_subwf
ModuleNotFoundError: No module named 'lib'
k

Kevin Su

10/06/2022, 8:54 PM
Hi @Frank Shen could you try
Copy code
pyflyte register --image dai/mlp/flyte/demo_image:1 refer_subworkflow_in_lib.py
f

Frank Shen

10/06/2022, 9:42 PM
Hi @Kevin Su I tried and got the same error:
Copy code
from lib.subwf import op, t1, my_subwf
ModuleNotFoundError: No module named 'lib'
k

Kevin Su

10/06/2022, 10:10 PM
It’s wired, mind sharing your entire folder to me (upload to github or somewhere). I can dig into it.
f

Frank Shen

10/06/2022, 10:11 PM
It’s not having a git repo yet. Will that be the issue?
Also I have flyte installed in the venv in the env folder.
k

Kevin Su

10/06/2022, 10:13 PM
it shouldn’t need to be git repo
f

Frank Shen

10/06/2022, 10:13 PM
and I have conda activated
I can try deactivate conda.
@Kevin Su, I found the reason. I have init file in the root folder. I don’t know why flyte doesn’t like it though. Do you?
After registering, I run the command below and it failed for ModuleNotFoundError: No module named ‘lib’ again in the pod.
Copy code
pyflyte run --remote refer_subworkflow_in_lib.py parent_wf --a 6
Do I need to restart the demo cluster in docker first?
Restarting demo cluster didn’t help.
k

Kevin Su

10/06/2022, 10:48 PM
you have to use pyflyte register, and manually trigger the workflow on flyte console
Copy code
pyflyte register refer_subworkflow_in_lib.py
y

Yee

10/06/2022, 10:58 PM
@Frank Shen the figuring out of where the base directory is is actually not trivial. the tools try to rely on the existence of init files to figure out where the base folder is.
of the given files, it walks up the folder structure until it finds a folder that doesn’t have an init file.
that folder is assumed to be the root. this doesn’t play nice with users who have namespace packages though (which is something we’re working on finding a ux-friendly way to address)
having init files set at the right place should get things to work
pyflyte run is different story - that’s only works if all your code is in a single file.
it was written as a quick and easy way to get started or iterate on things.
eventually as your repo grows you’ll want more than one file and you’ll have to move to the register/package patterns.
f

Frank Shen

10/06/2022, 11:01 PM
Thank you @Yee for the explanations!
4 Views