Thread
#flyte-deployment
    Nicholas LoFaso

    Nicholas LoFaso

    6 months ago
    Issue Two: Flytepropeller can no longer reach the flyte google bucket. It says
    Could not reach metadata service: Forbidden
    (rolling back to 0.19.2 the issue goes away)
    Full exception:
    [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    [ff161838c045048aab5a-n1-0] terminated with exit code (1). Reason [Error]. Message: 
    cmd())
      File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 463, in execute_task_cmd
        _execute_task(
      File "/usr/local/lib/python3.8/dist-packages/flytekit/common/exceptions/scopes.py", line 160, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 372, in _execute_task
        _handle_annotated_task(ctx, _task_def, inputs, output_prefix)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 283, in _handle_annotated_task
        _dispatch_execute(ctx, task_def, inputs, output_prefix)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py", line 179, in _dispatch_execute
        ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/core/data_persistence.py", line 418, in put_data
        raise FlyteAssertion(
    flytekit.common.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyteiye2s1de/local_flytekit/engine_dir to <gs://myflytebucket/metadata/propeller/nickflyte-dev-ff161838c045048aab5a/n1/data/0> (recursive=True).
    
    Original exception: Called process exited with error code: 1.  Stderr dump:
    
    b'Copying file:///tmp/flyteiye2s1de/local_flytekit/engine_dir/error.pb [Content-Type=application/octet-stream]...\nCould not reach metadata service: Forbidden\nCommandException: 1 file/object could not be transferred.\n/ [0/1 files][    0.0 B/  1.7 KiB]   0% Done                                    \r'
    .