rough-rose-81585
03/15/2022, 4:07 PMCould not reach metadata service: Forbidden
(rolling back to 0.19.2 the issue goes away)rough-rose-81585
03/15/2022, 4:08 PM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ff161838c045048aab5a-n1-0] terminated with exit code (1). Reason [Error]. Message:
cmd())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 463, in execute_task_cmd
_execute_task(
File "/usr/local/lib/python3.8/dist-packages/flytekit/common/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 372, in _execute_task
_handle_annotated_task(ctx, _task_def, inputs, output_prefix)
File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 283, in _handle_annotated_task
_dispatch_execute(ctx, task_def, inputs, output_prefix)
File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 179, in _dispatch_execute
ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)
File "/usr/local/lib/python3.8/dist-packages/flytekit/core/data_persistence.py", line 418, in put_data
raise FlyteAssertion(
flytekit.common.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyteiye2s1de/local_flytekit/engine_dir to <gs://myflytebucket/metadata/propeller/nickflyte-dev-ff161838c045048aab5a/n1/data/0> (recursive=True).
Original exception: Called process exited with error code: 1. Stderr dump:
b'Copying file:///tmp/flyteiye2s1de/local_flytekit/engine_dir/error.pb [Content-Type=application/octet-stream]...\nCould not reach metadata service: Forbidden\nCommandException: 1 file/object could not be transferred.\n/ [0/1 files][ 0.0 B/ 1.7 KiB] 0% Done \r'
.