Hi everyone.. any familiar with this error running...
# ask-the-community
e
Hi everyone.. any familiar with this error running a workflow? [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes. [a8crkvcv4mzm69kk5hp5-n0-0] terminated with exit code (1). Reason [Error]. Message: exec /opt/env/bin/pyflyte-execute: exec format error .
d
Hi @Eduardo Matus Can you please share the output of
kubectl get pods -n flyte
e
Hi @David Espejo (he/him), sorry for the late response. I create a new docker image and tested again with the same error. the output for the kubectl command is this: NAME READY STATUS RESTARTS AGE datacatalog-5d98867fcb-4l4bc 1/1 Running 0 47h datacatalog-5d98867fcb-fdj5v 1/1 Running 0 6d23h flyte-pod-webhook-6749b7d998-zsfrd 1/1 Running 0 5d1h flyteadmin-77586df548-6d6q2 1/1 Running 0 6d23h flyteadmin-77586df548-d6glf 1/1 Running 0 47h flyteconsole-577cfc8d9-8c9pv 1/1 Running 0 6d23h flyteconsole-577cfc8d9-fmmlg 1/1 Running 0 47h flytepropeller-646558445d-44pkk 1/1 Running 0 6d23h flytepropeller-646558445d-wwfr5 1/1 Running 0 47h flytescheduler-57cd6bff5b-q2pnz 1/1 Running 0 5d1h syncresources-6f65778494-9tqgc 1/1 Running 0 6d
the workflow run for 55 seconds but I don’t see the pod in the list aboce
above*
s
Could you fetch the pod log by running the following commands?
Copy code
kubectl get pods -n flytesnacks-development
kubectl logs <pod-name> -n flytesnacks-development
e
not much from the logs… kubectl logs a8crkvcv4mzm69kk5hp5-n0-0 -n flytesnacks-development exec /opt/env/bin/pyflyte-execute: exec format error
s
Can you
pip show flytekit
?
e
did another image, locally work, but running this on the aws cluster I get this: kubectl logs a9ftbgmzttd5xxhf29nq-n0-0 -n flytetester-staging Traceback (most recent call last): File “/opt/env/bin/pyflyte-execute”, line 8, in <module> sys.exit(execute_task_cmd()) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1130, in call return self.main(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1055, in main rv = self.invoke(ctx) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 760, in invoke return __callback(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 468, in execute_task_cmd _execute_task( File “/opt/env/lib/python3.9/site-packages/flytekit/exceptions/scopes.py”, line 160, in system_entry_point return wrapped(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 340, in _execute_task _task_def = resolver_obj.load_task(loader_args=resolver_args) File “/opt/env/lib/python3.9/site-packages/flytekit/core/python_auto_container.py”, line 194, in load_task task_module = importlib.import_module(task_module) File “/usr/local/lib/python3.9/importlib/__init__.py”, line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 972, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 972, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named ‘flyte’
d
can you share the workflow definition?
e
@David Espejo (he/him) Folder structure: . ├── Dockerfile ├── Dockerfile_old ├── LICENSE ├── README.md ├── pycache ├── aggs_requirements.txt ├── aws │ └── config ├── cn_universities.txt ├── docker_build_and_tag_old.sh ├── dockerbuild.sh ├── flyte-package.tgz ├── flyte.config ├── flytekit.config ├── requirements.txt └── workflows ├── init.py ├── pycache └── example.py Error: [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes. [abr42rr6whqllk2x5tqr-n0-0] terminated with exit code (1). Reason [Error]. Message: Traceback (most recent call last): File “/opt/env/bin/pyflyte-execute”, line 8, in <module> sys.exit(execute_task_cmd()) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1130, in call return self.main(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1055, in main rv = self.invoke(ctx) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 760, in invoke return __callback(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 468, in execute_task_cmd _execute_task( File “/opt/env/lib/python3.9/site-packages/flytekit/exceptions/scopes.py”, line 160, in system_entry_point return wrapped(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 340, in _execute_task _task_def = resolver_obj.load_task(loader_args=resolver_args) File “/opt/env/lib/python3.9/site-packages/flytekit/core/python_auto_container.py”, line 194, in load_task task_module = importlib.import_module(task_module) File “/usr/local/lib/python3.9/importlib/__init__.py”, line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 972, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named ‘workflows’ Workflow:
#!python import sys import typing from aggs_common.person_file_processor import person_file_processor from flytekit import task, workflow @task def say_hello() -> str: return “hello world” @workflow def my_wf() -> str: res = say_hello() return res if name == “__main__“: print(f”Running my_wf() { my_wf() }“)
packaging and registration: pyflyte --pkgs workflows package --image $AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG -f echo “flyte register” flytectl register files flyte-package.tgz -p flyteexamples -d staging --version 1.6 --archive -c ~/.flyte/config.yaml
s
Can you move
workflows
to a parent directory that has an
__init__.py
file?
e
@Samhita Alla thanks, I added the init.py inside the workflows folder didn’t work. the error was “no module named workflows”. Then added a new
__init__.py
in the parent folder and the error now is “no module named dockerization” This is my folder structure
Copy code
dockerization
├── dockerization/Dockerfile
├── dockerization/Dockerfile_old
├── dockerization/LICENSE
├── dockerization/README.md
├── dockerization/__init__.py
├── dockerization/aggs_requirements.txt
├── dockerization/aws
│   └── dockerization/aws/config
├── dockerization/docker_build_and_tag_old.sh
├── dockerization/dockerbuild.sh
├── dockerization/flyte.config
├── dockerization/flytekit.config
├── dockerization/requirements.txt
└── dockerization/workflows
    ├── dockerization/workflows/__init__.py    
    └── dockerization/workflows/example.py
Error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f138ab0c8b09140fab80-n0-0] terminated with exit code (1). Reason [Error]. Message: 
>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dockerization'
s
Can you add
__init__.py
to
dockerization
folder?
e
I already have it, the error above is with the
__init__.py
in the dockerization folder
s
I'm assuming
workflows
is present in
dockerization
, right? Can you add, say,
parent_worfklows
folder in
dockerization
and move your
workflows
folder to
parent_workflows
? The hierarchy has to be:
dockerization
>
parent_workflows
>
workflows
.
parent_workfows
and
workflows
folders need to have
__init__.py
files.
e
will do. an
__init__.py
is needed on each folder or only in the workflows folder?
s
workflows
&
parent_workflows
. Remove it in
dockerization
.
e
ok
s
If this doesn't work, please share with me a UI screen grab and the registration command you're using.
e
@Samhita Alla Didn’t work. Here is how I push the image, create the package and register
Copy code
echo "pushing image"
docker push $<http://AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG|AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG>
echo "flyte package"
pyflyte --pkgs parent_workflows.workflows package --image $<http://AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG|AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG>  -f
echo "flyte register"
flytectl -c ~/.flyte/config.yaml register files flyte-package.tgz -p flyteexamples -d production --version 1.6 --archive
echo "run"
pyflyte -c ~/.flyte/config.yaml run --remote --image $<http://AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG|AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG> parent_workflows.workflows.example.py my_wf
Also, this is the full error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[az89hdtf99phv6744gn6-n0-0] terminated with exit code (1). Reason [Error]. Message: 
/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/env/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 476, in execute_task_cmd
    _execute_task(
  File "/opt/env/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/opt/env/lib/python3.9/site-packages/flytekit/core/python_auto_container.py", line 280, in load_task
    task_module = importlib.import_module(task_module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'parent_workflows'
.
Folder structure:
Copy code
dockerization
├── dockerization/LICENSE
├── dockerization/README.md
├── dockerization/flyte.config
├── dockerization/flytekit.config
└── dockerization/parent_workflows
    ├── dockerization/parent_workflows/__init__.py
    └── dockerization/parent_workflows/workflows
        ├── dockerization/parent_workflows/workflows/__init__.py
        └── dockerization/parent_workflows/workflows/example.py
s
@Eduardo Matus, it works for me. I followed the same directory structure and ran the workflow using
pyflyte run --remote parent_workflows.workflows.example.py my_wf --a 1 --b "hello"
command. You needn't package and register separately if you're using
pyflyte run
cause it packages and registers the tasks and workflows. ── dockerization │ └── parent_workflows │ ├── init.py │ └── workflows │ ├── init.py │ └── example.py
e
Hi @Samhita Alla. running the workflow with the remote flag without the container works fine. The problem happen when using an image.
s
Are you running
pyflyte run
without running the other two commands?
Even if you specify an image,
pyflyte run
will not use the code that's present in the image.
So you shouldn't be seeing any module not found error. Are you looking at the execution that's triggered by
pyflyte run
?
e
I don’t get module not found errors unless and I add a library like html2text, I get
ModuleNotFoundError: No module named 'html2text'
@Samhita Alla Thanks for the help
s
Oh. You shouldn't be seeing that error if you install the library in the Docker image.
e
running this
pyflyte run --remote parent_workflows.workflows.example.py my_wf --a 1 --b "hello"
with html2text and without a container should fail, right?
s
Yep!
e
and how I do run the container in the cluster? you mentioned earlier that doing
pyflyte run --remote --image container_image
doesn’t execute the container code
s
I didn't get you. In fast registration, having the necessary libraries in the container image should suffice. The code is packaged and zipped up and is available to Flyte during execution. Are you still seeing the module not found error?
e
@Samhita Alla All working now.. thanks for help and your time!
196 Views