Hi everyone.. any familiar with this error running a workflow? [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes. [a8crkvcv4mzm69kk5hp5-n0-0] terminated with exit code (1). Reason [Error]. Message: exec /opt/env/bin/pyflyte-execute: exec format error .

Hi @rhythmic-lizard-91902 Can you please share the output of

kubectl get pods -n flyte

Hi @average-finland-92144, sorry for the late response. I create a new docker image and tested again with the same error. the output for the kubectl command is this: NAME READY STATUS RESTARTS AGE datacatalog-5d98867fcb-4l4bc 1/1 Running 0 47h datacatalog-5d98867fcb-fdj5v 1/1 Running 0 6d23h flyte-pod-webhook-6749b7d998-zsfrd 1/1 Running 0 5d1h flyteadmin-77586df548-6d6q2 1/1 Running 0 6d23h flyteadmin-77586df548-d6glf 1/1 Running 0 47h flyteconsole-577cfc8d9-8c9pv 1/1 Running 0 6d23h flyteconsole-577cfc8d9-fmmlg 1/1 Running 0 47h flytepropeller-646558445d-44pkk 1/1 Running 0 6d23h flytepropeller-646558445d-wwfr5 1/1 Running 0 47h flytescheduler-57cd6bff5b-q2pnz 1/1 Running 0 5d1h syncresources-6f65778494-9tqgc 1/1 Running 0 6d

Could you fetch the pod log by running the following commands?

kubectl get pods -n flytesnacks-development
kubectl logs <pod-name> -n flytesnacks-development

not much from the logs… kubectl logs a8crkvcv4mzm69kk5hp5-n0-0 -n flytesnacks-development exec /opt/env/bin/pyflyte-execute: exec format error

Can you

pip show flytekit

?

did another image, locally work, but running this on the aws cluster I get this: kubectl logs a9ftbgmzttd5xxhf29nq-n0-0 -n flytetester-staging Traceback (most recent call last): File “/opt/env/bin/pyflyte-execute”, line 8, in <module> sys.exit(execute_task_cmd()) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1130, in call return self.main(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1055, in main rv = self.invoke(ctx) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 760, in invoke return __callback(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 468, in execute_task_cmd _execute_task( File “/opt/env/lib/python3.9/site-packages/flytekit/exceptions/scopes.py”, line 160, in system_entry_point return wrapped(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 340, in _execute_task _task_def = resolver_obj.load_task(loader_args=resolver_args) File “/opt/env/lib/python3.9/site-packages/flytekit/core/python_auto_container.py”, line 194, in load_task task_module = importlib.import_module(task_module) File “/usr/local/lib/python3.9/importlib/__init__.py”, line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 972, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 972, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named ‘flyte’

can you share the workflow definition?

@average-finland-92144 Folder structure: . ├── Dockerfile ├── Dockerfile_old ├── LICENSE ├── README.md ├── pycache ├── aggs_requirements.txt ├── aws │ └── config ├── cn_universities.txt ├── docker_build_and_tag_old.sh ├── dockerbuild.sh ├── flyte-package.tgz ├── flyte.config ├── flytekit.config ├── requirements.txt └── workflows ├── init.py ├── pycache └── example.py Error: [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes. [abr42rr6whqllk2x5tqr-n0-0] terminated with exit code (1). Reason [Error]. Message: Traceback (most recent call last): File “/opt/env/bin/pyflyte-execute”, line 8, in <module> sys.exit(execute_task_cmd()) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1130, in call return self.main(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1055, in main rv = self.invoke(ctx) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File “/opt/env/lib/python3.9/site-packages/click/core.py”, line 760, in invoke return __callback(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 468, in execute_task_cmd _execute_task( File “/opt/env/lib/python3.9/site-packages/flytekit/exceptions/scopes.py”, line 160, in system_entry_point return wrapped(*args, **kwargs) File “/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py”, line 340, in _execute_task _task_def = resolver_obj.load_task(loader_args=resolver_args) File “/opt/env/lib/python3.9/site-packages/flytekit/core/python_auto_container.py”, line 194, in load_task task_module = importlib.import_module(task_module) File “/usr/local/lib/python3.9/importlib/__init__.py”, line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 972, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load File “<frozen importlib._bootstrap>“, line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named ‘workflows’ Workflow:

Can you move

workflows

to a parent directory that has an

__init__.py

file?

@tall-lock-23197 thanks, I added the init.py inside the workflows folder didn’t work. the error was “no module named workflows”. Then added a new

__init__.py

in the parent folder and the error now is “no module named dockerization” This is my folder structure

dockerization
├── dockerization/Dockerfile
├── dockerization/Dockerfile_old
├── dockerization/LICENSE
├── dockerization/README.md
├── dockerization/__init__.py
├── dockerization/aggs_requirements.txt
├── dockerization/aws
│   └── dockerization/aws/config
├── dockerization/docker_build_and_tag_old.sh
├── dockerization/dockerbuild.sh
├── dockerization/flyte.config
├── dockerization/flytekit.config
├── dockerization/requirements.txt
└── dockerization/workflows
    ├── dockerization/workflows/__init__.py    
    └── dockerization/workflows/example.py

Error:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f138ab0c8b09140fab80-n0-0] terminated with exit code (1). Reason [Error]. Message: 
>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dockerization'

Can you add

__init__.py

to

dockerization

folder?

I already have it, the error above is with the

__init__.py

in the dockerization folder

I'm assuming

workflows

is present in

dockerization

, right? Can you add, say,

parent_worfklows

folder in

dockerization

and move your

workflows

folder to

parent_workflows

? The hierarchy has to be:

dockerization

>

parent_workflows

>

workflows

.

parent_workfows

and

workflows

folders need to have

__init__.py

files.

will do. an

__init__.py

is needed on each folder or only in the workflows folder?

workflows

&

parent_workflows

. Remove it in

dockerization

.

ok

If this doesn't work, please share with me a UI screen grab and the registration command you're using.

@tall-lock-23197 Didn’t work. Here is how I push the image, create the package and register

echo "pushing image"
docker push $<http://AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG|AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG>
echo "flyte package"
pyflyte --pkgs parent_workflows.workflows package --image $<http://AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG|AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG>  -f
echo "flyte register"
flytectl -c ~/.flyte/config.yaml register files flyte-package.tgz -p flyteexamples -d production --version 1.6 --archive
echo "run"
pyflyte -c ~/.flyte/config.yaml run --remote --image $<http://AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG|AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/$IMAGE_TAG> parent_workflows.workflows.example.py my_wf

Also, this is the full error:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[az89hdtf99phv6744gn6-n0-0] terminated with exit code (1). Reason [Error]. Message: 
/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/env/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 476, in execute_task_cmd
    _execute_task(
  File "/opt/env/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/env/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/opt/env/lib/python3.9/site-packages/flytekit/core/python_auto_container.py", line 280, in load_task
    task_module = importlib.import_module(task_module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'parent_workflows'
.

@rhythmic-lizard-91902, it works for me. I followed the same directory structure and ran the workflow using

pyflyte run --remote parent_workflows.workflows.example.py my_wf --a 1 --b "hello"

command. You needn't package and register separately if you're using

pyflyte run

cause it packages and registers the tasks and workflows. ── dockerization │ └── parent_workflows │ ├── init.py │ └── workflows │ ├── init.py │ └── example.py

Hi @tall-lock-23197. running the workflow with the remote flag without the container works fine. The problem happen when using an image.

Are you running

pyflyte run

without running the other two commands?

I don’t get module not found errors unless and I add a library like html2text, I get

ModuleNotFoundError: No module named 'html2text'

Oh. You shouldn't be seeing that error if you install the library in the Docker image.

running this

pyflyte run --remote parent_workflows.workflows.example.py my_wf --a 1 --b "hello"

with html2text and without a container should fail, right?

Yep!

and how I do run the container in the cluster? you mentioned earlier that doing

pyflyte run --remote --image container_image

doesn’t execute the container code

I didn't get you. In fast registration, having the necessary libraries in the container image should suffice. The code is packaged and zipped up and is available to Flyte during execution. Are you still seeing the module not found error?

@tall-lock-23197 All working now.. thanks for help and your time!