acoustic-carpenter-78188
03/01/2023, 8:53 AM<https://www.googleapis.com/auth/devstorage.read_write> seems to be no longer enough in GKE versions >=1.25.0. We could not find any release notes from Google pointing in this direction but noticed that Flyte would no longer work after we updated the GKE cluster version to 1.25.5.
Fast registration failed with this error response from `flyteadmin`:
..
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/remote/remote.py", line 580, in fast_package
return self._upload_file(pathlib.Path(zip_file))
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/remote/remote.py", line 598, in _upload_file
upload_location = self.client.get_upload_signed_url(
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
return super(SynchronousFlyteClient, self).create_upload_location(
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/raw.py", line 41, in handler
return fn(*args, **kwargs)
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/raw.py", line 856, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.
Details:
[
{
"@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
"domain": "<http://googleapis.com|googleapis.com>",
"metadata": {
"method": "google.iam.credentials.v1.IAMCredentials.SignBlob",
"service": "<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>"
},
"reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
}
]"
debug_error_string = "UNKNOWN:Error received from peer ipv4:{removed} {created_time:"2023-02-20T11:01:31.243469439+00:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\nDetails:\n[\n {\n \"@type\": \"<http://type.googleapis.com/google.rpc.ErrorInfo\|type.googleapis.com/google.rpc.ErrorInfo\>",\n \"domain\": \"<http://googleapis.com|googleapis.com>\",\n \"metadata\": {\n \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\n \"service\": \"<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>\"\n },\n \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\n }\n]"}"
Check all the applicable boxes
☐ I updated the documentation accordingly. (no documentation needs to be updated?)
☑︎ All new and existing tests passed. (I ran the helm and kustomize test targets)
☑︎ All commits are signed-off.
Note to reviewers
We started a discussion in Slack on this issue. @thankful-minister-83577 .
As far as we know there is no more restrictive scope available. We checked this Google page and it seems like google.iam.credentials.v1.IAMCredentials.SignBlob is only available in the cloud-platform scope.
We also tried <https://www.googleapis.com/auth/devstorage.full_control> and it was not enough.
flyteorg/flyte
✅ All checks have passed
2/2 successful checks