boundless-pizza-95864
03/07/2022, 7:56 PM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[vej152kn4b-n1-0] terminated with exit code (1). Reason [Error]. Message:
tar: /root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Traceback (most recent call last):
File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 467, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/opt/venv/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 112, in download_distribution
result.check_returncode()
File "/usr/lib/python3.9/subprocess.py", line 460, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tar', '-xvf', '/root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz', '-C', '/root']' returned non-zero exit status 2.
.
Here's what's executing from the pod spec
spec:
containers:
- args:
- pyflyte-fast-execute
- --additional-distribution
- <gs://my-bucket/fast/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz>
...
I checked the bucket and the object exists so uploading during registration seems to be working correctly. Perhaps something wrong with the download or the path it is copied to in the container. Any ideas?freezing-airport-6809
thankful-minister-83577
thankful-minister-83577
boundless-pizza-95864
03/07/2022, 9:00 PMboundless-pizza-95864
03/07/2022, 9:03 PMthankful-minister-83577
thankful-minister-83577
boundless-pizza-95864
03/07/2022, 9:06 PMboundless-pizza-95864
03/07/2022, 9:07 PMfreezing-airport-6809
freezing-airport-6809
thankful-minister-83577
thankful-minister-83577
boundless-pizza-95864
03/07/2022, 9:16 PM[tool.poetry.dependencies]
python = "^3.9"
flytekit = "^0.30.3"
flytekitplugins-data-fsspec = "^0.30.3"
gcsfs = "^2022.1.0"
...
freezing-airport-6809
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
boundless-pizza-95864
03/07/2022, 10:03 PMboundless-pizza-95864
03/07/2022, 10:15 PMthankful-minister-83577
boundless-pizza-95864
03/08/2022, 3:03 PMthankful-minister-83577