Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by LinenTitle
a
Armaan Goel
09/17/2022, 4:31 PMHaving trouble running
pyflyte run --remote core/flyte_basics/hello_world.py my_wfk
Ketan (kumare3)
09/17/2022, 4:45 PMWhat's the error
We need to
Improve error messages for Grpc
a
Armaan Goel
09/17/2022, 6:53 PMDang i lost the original error trying to debug this
I think the core thing I’m confused about is
here in the tutorial https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ssl-certificate
It talks about how to set up google managed ssl cert, but it seems like we can’t actually use that without gke ingress and the rest of the tutorial uses nginx-ingress?
When I try the nginx/cert-manager way that is described in the tutorial that doesnt work
(`
helm install cert-manager --namespace flyte --version v0.12.0 jetstack/cert-managerError: failed to install CRD crds/certificaterequests.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "<http://apiextensions.k8s.io/v1beta1|apiextensions.k8s.io/v1beta1>"Ok, got past that issue (happy to share what I did), but getting this error now
details = "failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.
Details:
[
{
"@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
"domain": "<http://googleapis.com|googleapis.com>",
"metadata": {
"method": "google.iam.credentials.v1.IAMCredentials.SignBlob",
"service": "<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>"
},
"reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
}
]"k
Ketan (kumare3)
09/18/2022, 4:34 AMI think the service account for FlyteAdmin does not have right Iam perms
a
Armaan Goel
09/18/2022, 4:35 AMI added the the permission to FlyteAdmin as instructed actually
In fact I added it to all Flyte roles to see if that helps
k
Ketan (kumare3)
09/18/2022, 4:35 AMThis is only admin
Seems right
Did you use workload identity to attach
a
Armaan Goel
09/18/2022, 6:02 AMIn case it is helpful, here’s what I did to fix the earlier SSL problem: https://docs.google.com/document/d/1skJWmt3hJoIuPQr_RfR-gB9wlatVSIcSD5VlBylJqd8/edit?usp=sharing
k
Ketan (kumare3)
09/18/2022, 8:26 PMCc @Smriti Satyan can we doc this
a
Armaan Goel
09/19/2022, 6:29 PMIf anyone had an advice on fixing these IAM issues, I’d really appreciate it! Ive tried a lot of different things
k
Ketan (kumare3)
09/19/2022, 6:41 PMOhh sorry, cc @jeev do you know
j
jeev
09/19/2022, 6:44 PMhmm not sure. @Armaan Goel: where was the
ACCESS_TOKEN_SCOPE_INSUFFICIENTflyte shouldnt be using signed urls for gcp.
do you have this config for admin, propeller, and data catalog?
storage:
type: stow
stow:
kind: google
config:
json: ""
# replace with the GCP project ID
project_id:
scopes: <https://www.googleapis.com/auth/devstorage.read_write>k
Ketan (kumare3)
09/19/2022, 7:07 PM@jeev infact we do, this is for pyflyte run
j
jeev
09/19/2022, 7:09 PMwould that make a difference @Ketan (kumare3)?
going based off of this error:
details = "failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.it shouldnt have to create signed urls, right?
im curious where this error message was observed. was it in the flyte control plane or locally when running the
pyflyte runa
Armaan Goel
09/19/2022, 7:14 PMI think it was in the control plane it looked like a remote error that was just communicated by grpc
Let me find the exact error
armaangoel78@cloudshell:~/Gauntlet/flytesnacks/cookbook (gauntletai)$ pyflyte run --remote core/flyte_basics/hello_world.py my_wf
{"asctime": "2022-09-19 19:16:13,698", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = \"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\nDetails:\n[\n {\n \"@type\": \"<http://type.googleapis.com/google.rpc.ErrorInfo\|type.googleapis.com/google.rpc.ErrorInfo\>",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\n \"service\": \"iamcredentials.googleapis.com\"\n },\n \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\n }\n]\"\n\tdebug_error_string = \"{\"created\":\"@1663614973.698578904\",\"description\":\"Error received from peer ipv4:35.199.161.247:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":966,\"grpc_message\":\"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\\nDetails:\\n[\\n {\\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\\n \"domain\": \"<http://googleapis.com|googleapis.com>\",\\n \"metadata\": {\\n \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\\n \"service\": \"<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>\"\\n },\\n \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\\n }\\n]\",\"grpc_status\":2}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-09-19 19:16:13,917", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = \"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\nDetails:\n[\n {\n \"@type\": \"<http://type.googleapis.com/google.rpc.ErrorInfo\|type.googleapis.com/google.rpc.ErrorInfo\>",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\n \"service\": \"iamcredentials.googleapis.com\"\n },\n \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\n }\n]\"\n\tdebug_error_string = \"{\"created\":\"@1663614973.917603494\",\"description\":\"Error received from peer ipv4:35.199.161.247:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":966,\"grpc_message\":\"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\\nDetails:\\n[\\n {\\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\\n \"domain\": \"<http://googleapis.com|googleapis.com>\",\\n \"metadata\": {\\n \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\\n \"service\": \"<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>\"\\n },\\n \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\\n }\\n]\",\"grpc_status\":2}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
File "/home/armaangoel78/.local/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 539, in _run
remote_entity = remote.register_script(
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/remote/remote.py", line 596, in register_script
upload_location, md5_bytes = fast_register_single_script(
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 113, in fast_register_single_script
upload_location = create_upload_location_fn(content_md5=md5)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
return super(SynchronousFlyteClient, self).create_upload_location(
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 41, in handler
return fn(*args, **kwargs)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 854, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/armaangoel78/.local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.
Details:
[
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"domain": "googleapis.com",
"metadata": {
"method": "google.iam.credentials.v1.IAMCredentials.SignBlob",
"service": "iamcredentials.googleapis.com"
},
"reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
}
]"
debug_error_string = "{"created":"@1663614974.333626746","description":"Error received from peer ipv4:35.199.161.247:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\nDetails:\n[\n {\n "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n "domain": "googleapis.com",\n "metadata": {\n "method": "google.iam.credentials.v1.IAMCredentials.SignBlob",\n "service": "<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>"\n },\n "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"\n }\n]","grpc_status":2}"
>Also @jeev how did u get that view for the admin, propeller, etc config?
I just see a list of permissions on the console at least
j
jeev
09/19/2022, 7:22 PMit should be in the respective kubernetes configmaps
you can go to Kubernetes Engine > Secrets and ConfigMaps and browse there
a
Armaan Goel
09/19/2022, 7:23 PMOh ill take a look
j
jeev
09/19/2022, 7:23 PMor just do
kubectl get configmap -n flyte <name of configmap> -o yamlthis looks like the likely source:
File "/home/armaangoel78/.local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 854, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)k
Ketan (kumare3)
09/19/2022, 7:27 PM@jeev the error is returned from the FlyteAdmin.
Flyteadmin tries to sign
so my guess is for some reason the permissions for GCP / GCS are inadequate
just do not know how - cc @Haytham Abuelfutuh
j
jeev
09/19/2022, 7:28 PMyou probably just need additional permissions to sign the urls: https://cloud.google.com/storage/docs/access-control/signing-urls-manually#prereqs
see the starred note
TIL about pyflyte run 😛
actually the above permissions look like they already encompass that
@Armaan Goel: these look like custom roles in GCP. are these roles attached to the serviceaccount bound to the flyte components?
h
Haytham Abuelfutuh
09/19/2022, 7:36 PM@Armaan Goel Sorry you are having trouble with this! Signed URLs on GCS definitely work so let’s figure out where configs went wrong (and then update docs to tell people exactly what they need 😄)
To cover our basis, let’s try canned examples first:
flytectl config init --host <flyte url>
flytectl register examples -p flytesnacks -d developmenta
Armaan Goel
09/19/2022, 9:25 PMHey @jeev
yeah i thinking they are attached to the gcp roles using workload identity
Hmm different error with that Haytham:
----------------------------------------------------------------------------------------------- -------- -------------------------------------------------------
| NAME | STATUS | ADDITIONAL INFO |
----------------------------------------------------------------------------------------------- -------- -------------------------------------------------------
| /tmp/register1646303769/snacks-cookbook-case_studies-bioinformatics-blast/0__bash.blastx_1.pb | Failed | Error registering file due to rpc error: code = |
| | | Unavailable desc = name resolver error: produced zero |
| | | addresses |
----------------------------------------------------------------------------------------------- -------- -------------------------------------------------------
1 rows
Error: example 0xc0007b0ab0 failed to register rpc error: code = Unavailable desc = name resolver error: produced zero addressesj
jeev
09/19/2022, 10:04 PMim starting to doubt workload identity or just plain permissions assignment more
a
Armaan Goel
09/19/2022, 10:12 PMhmm
so sorry to add some more confusion to them mix, but I’m starting from scratch on a brand new project
But I’m wondering, does this line
gcloud iam service-accounts add-iam-policy-binding --role "roles/iam.workloadIdentityUser" --member "serviceAccount:${PROJECT_ID}.svc.id.goog[flyte/flyteadmin]" gsa-flyteadmin@${PROJECT_ID}.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>need to be run after the cluster is created?
this is from here https://docs.flyte.org/en/latest/deployment/gcp/manual.html#permissions
It errors, and I dont think it can be run because we havent created a cluster yet
j
jeev
09/20/2022, 2:46 AMwait if you dont have a cluster, where is flyte running?
a
Armaan Goel
09/20/2022, 4:43 AMsorry yeah im confusing things by restarting
all of the stuff I showed earlier is running on a flyte cluster,
i just started a new project, where I am following the tutorail from scratch, so i havent gotten to the cluster creation step yet, and i have this seperate issue
WOOOOOHOOOO I got it!
I’ll document what I did in case people run into the same thing in the future
but basically yes you do need to turn on workload identity for the cluster, but you also need to turn on “GKE Metadata Server” for the individual node pools (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#option_2_node_pool_modification)
Really appreciate everyone lending a hand to take a look here!
j
jeev
09/20/2022, 5:46 AMglad to hear it worked out @Armaan Goel!