Sören Brunk
03/07/2022, 7:56 PM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[vej152kn4b-n1-0] terminated with exit code (1). Reason [Error]. Message:
tar: /root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Traceback (most recent call last):
File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 467, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/opt/venv/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 112, in download_distribution
result.check_returncode()
File "/usr/lib/python3.9/subprocess.py", line 460, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tar', '-xvf', '/root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz', '-C', '/root']' returned non-zero exit status 2.
.
Here's what's executing from the pod spec
spec:
containers:
- args:
- pyflyte-fast-execute
- --additional-distribution
- <gs://my-bucket/fast/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz>
...
I checked the bucket and the object exists so uploading during registration seems to be working correctly. Perhaps something wrong with the download or the path it is copied to in the container. Any ideas?Ketan (kumare3)
Yee
Sören Brunk
03/07/2022, 9:00 PMYee
Sören Brunk
03/07/2022, 9:06 PMKetan (kumare3)
Yee
Sören Brunk
03/07/2022, 9:16 PM[tool.poetry.dependencies]
python = "^3.9"
flytekit = "^0.30.3"
flytekitplugins-data-fsspec = "^0.30.3"
gcsfs = "^2022.1.0"
...
Ketan (kumare3)
Yee
Sören Brunk
03/07/2022, 10:03 PMYee
Sören Brunk
03/08/2022, 3:03 PMYee