https://flyte.org logo
Join Slack
Powered by
Hi! Noob question here. I'm trying to use `ImageSp...
# ask-the-community
a
Hi! Noob question here. I'm trying to use 
ImageSpec
in a given task I am getting a 
ModuleNotFoundError
on the parent directory, even on the hello_world example, whenever I run the workflow 
--remote
. Does this have to do with how pyflyte packages the workflow? How should I set up the tree? Example workflow:
Copy code
from flytekit import task, workflow, ImageSpec

image_spec = ImageSpec(
    name="helloworld",
    base_image="python:3.11.5-slim-bullseye",
    packages=["numpy", "pandas", "scipy", "scikit-learn", "sentence_transformers", "k_means_constrained", "flytekit"],
    registry="localhost:30000"
)

@task(container_image=image_spec)
def say_hello() -> str:
    return "Hello, World!"

@workflow
def hello_world_wf() -> str:
    res = say_hello()
    return res

if __name__ == "__main__":
    print(f"Running hello_world_wf() {hello_world_wf()}")
Run with 
pyflyte run --remote parent_directory_workflows/workflows/hello_world.py hello_world_wf
, yields 
ModuleNotFoundError: No module named 'parent_directory_workflows'
, with tree being:
Copy code
.
├── README.md
├── parent_directory_workflows
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-311.pyc
│   └── workflows
│       ├── __init__.py
│       ├── __pycache__
│       │   ├── __init__.cpython-311.pyc
│       │   └── hello_world.cpython-311.pyc
│       └── hello_world.py
├── poetry.lock
└── pyproject.toml
k
could you try default base image?
Copy code
image_spec = ImageSpec(
    name="helloworld",
    packages=["numpy", "pandas", "scipy", "scikit-learn", "sentence_transformers", "k_means_constrained", "flytekit"],
    registry="localhost:30000"
)
investigating in the meantime
a
Default base image succeded! What is the issue with other base images as bullseye then?
As an additional point, I see that if I run again 
pyflyte run --remote parent_directory_workflows/workflows/hello_world.py hello_world_wf
it doesn't find the image built a minute ago and supposedly pushed to 
localhost:30000
, so it builds again... any idea why?
k
since you change the default image, it should rebuild it right?
a
The first time after the change yes, but it does it a third time if I pyflyte run without changing the image again
k
ah, i see. that’s not correct. which version of flytekit are you using
a
1.12.0, installed with poetry
k
it doesn’t rebuild for me
does image url change after you rerun the task?
a
No, it doesn't
However, interestingly if I tag the image with
Copy code
image_spec = ImageSpec(
    name="helloworld",
    tag_format="devtest",
    packages=["numpy", "pandas", "scipy", "scikit-learn", "sentence_transformers", "k_means_constrained", "flytekit"],
    registry="localhost:30000"
)
I get an error, so I'm thinking the push is actually not working
Copy code
Failed with Unknown Exception <class 'Exception'> Reason: failed to run command envd build --path /var/folders/21/2_6x0r111jb306csghn5w4hm0000gp/T/flytev2pnykln/control_plane_metadata/local_flytekit/54adc7d84fb04862e495d40d5b2c8916  --platform linux/amd64 --output type=image,name=localhost:30000/helloworld:devtest,push=true with error b'time="2024-05-29T09:43:17+02:00" level=fatal msg=exit app=envd error="failed to build the image: failed to build: failed to wait error group: failed to solve LLB: failed to solve: failed to push localhost:30000/helloworld:devtest: failed to do request: Head \\"<http://localhost:30000/v2/helloworld/blobs/sha256:72914424168c8ebb0dbb3d0e08eb1d3b5b2a64cc51745bd65caf29c335b31dc7>\\": dial tcp 127.0.0.1:30000: connect: connection refused" version=v0.3.45\n'
k
I have a fix for base image issue btw. https://github.com/flyteorg/flytekit/pull/2447
could you show me the output of running 
pyflyte run
two times in the terminal?
Trying to reproduce that issue
a
Sure, this is for the first run (I reduced the installed packages for testing speed)
run_1.txt
While this is the second run
run_2.txt
The same happens if I pyflyte run twice while tagging the image
Copy code
Running Execution on Remote.
Image localhost:30000/helloworld:devtest not found. building...
Run command: envd context use --name flyte-sandbox 
Run command: envd build --path /var/folders/21/2_6x0r111jb306csghn5w4hm0000gp/T/flyter9pdheml/control_plane_metadata/local_flytekit/baf0ee0c705c6a72578a743ba28b4ce8  --platform linux/amd64 --output type=image,name=localhost:30000/helloworld:devtest,push=true 
#1 <docker-image://cr.flyte.org/flyteorg/flytekit:py3.11-1.12.0>
#1 resolve cr.flyte.org/flyteorg/flytekit:py3.11-1.12.0 0.7s done
#1 DONE 0.7s
#2 [internal] setting pip cache mount permissions
#2 CACHED
#3 [internal] language environment and system packages
#3 CACHED
#4 [internal] setting PyPI index dir /etc
#4 CACHED
#5 [internal] setting PyPI index file /etc/pip.conf
#5 CACHED
#6 [internal] prepare language
#6 CACHED
#7 [internal] create cache dir
#7 CACHED
#8 [internal] pip install flytekit numpy regex
#8 CACHED
#9 bash -c set -euo pipefail
#9 DONE 0.3s
#10 exporting to image
#10 exporting layers 0.0s done
#10 exporting manifest sha256:71aa3fdfdb017c14f2246951c7d695d53a982ddca2ef9451e6ada0bc255bbec3 done
#10 exporting config sha256:7116af0e88a20063cd54be16cb70fde55be65cc49dee2e8c3229a3178ccf8af1 done
#10 pushing layers 0.0s done
#10 pushing manifest for localhost:30000/helloworld:devtest@sha256:71aa3fdfdb017c14f2246951c7d695d53a982ddca2ef9451e6ada0bc255bbec3 done
#10 DONE 0.1s

[✔] Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f5d6260598b564009b18> to see execution in the console.
and
Copy code
Running Execution on Remote.
Image localhost:30000/helloworld:devtest not found. building...
Run command: envd context use --name flyte-sandbox 
Run command: envd build --path /var/folders/21/2_6x0r111jb306csghn5w4hm0000gp/T/flytejvizudc5/control_plane_metadata/local_flytekit/46898132c51e107232f282bb503689ba  --platform linux/amd64 --output type=image,name=localhost:30000/helloworld:devtest,push=true 
#1 <docker-image://cr.flyte.org/flyteorg/flytekit:py3.11-1.12.0>
#1 resolve cr.flyte.org/flyteorg/flytekit:py3.11-1.12.0 0.7s done
#1 DONE 0.7s
#2 [internal] setting pip cache mount permissions
#2 CACHED
#3 [internal] setting PyPI index dir /etc
#3 CACHED
#4 [internal] setting PyPI index file /etc/pip.conf
#4 CACHED
#5 [internal] prepare language
#5 CACHED
#6 [internal] language environment and system packages
#6 CACHED
#7 [internal] create cache dir
#7 CACHED
#8 [internal] pip install flytekit numpy regex
#8 CACHED
#9 bash -c set -euo pipefail
#9 DONE 0.2s
#10 exporting to image
#10 exporting layers 0.0s done
#10 exporting manifest sha256:d1b3efbf2a4e09f014b7f2459eb587f354afcc055be9fcdc49d6d2d1e3467f5c done
#10 exporting config sha256:bb4351daa30f2eb2b8279c7119899508480850bdd38316ff30cc873ca2c2ee5f done
#10 pushing layers 0.0s done
#10 pushing manifest for localhost:30000/helloworld:devtest@sha256:d1b3efbf2a4e09f014b7f2459eb587f354afcc055be9fcdc49d6d2d1e3467f5c
#10 pushing manifest for localhost:30000/helloworld:devtest@sha256:d1b3efbf2a4e09f014b7f2459eb587f354afcc055be9fcdc49d6d2d1e3467f5c done
#10 DONE 0.1s

[✔] Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f15690c2df320419fb65> to see execution in the console.
Hi @Kevin Su, just to add on this: if I try to 
docker pull localhost:30000/helloworld:devtest
docker finds and pulls the image, however 
pyflyte run
doesn't finds it and yields 
Image localhost:30000/helloworld:devtest not found. building...
- so the issue seems to be finding the image, pushing to localhost from pyflyte is working
Hi Kevin, I managed to get this working - not sure where the communication problem was, I think in how my venv was communicating with the cluster - tore down and reinstalled flytectl and docker and recreated the venv. Thanks for your time and support!
16 Views
Open in Slack
PreviousNext
Powered by