Ariel Kaspit
05/31/2023, 4:41 PMseparateGrpcIngress
? I understand from the documentation that it is Required for certain ingress controllers like nginx.
2. After deploying nginx (which uses our own tls certificate, not self-signed with cert-manager), I’m getting 502 bad gateway errors accessing the console (flyte.my.domain)… In the flyteadmin
logs I see this authentication error: Failed to refresh tokens. Restarting login flow. Error: [TOKEN_REFRESH_FAILURE] Error refreshing token, caused by: oauth2: cannot fetch token: 400 Bad Request
(it worth mentioning I don’t see any errors in Okta, which we configured there the authorization server)
3. Regarding accessing within the CLI, while trying to use flytectl
I’m still getting the same authentication error: PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
- Did you experience this thing? BTW, I don’t understand from your answer if you’re using flytectl/pyflyte
?
Thanks in advanced! And sorry for all the questions, I appreciate your help!pyflyte
and the error is:
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Received http2 header with status: 403"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Received http2 header with status: 403", grpc_status:7, created_time:"2023-05-31T19:41:03.999452+03:00"}"
It seems to still have issues with the gRPC service… Did I miss something?flyteadmin
and datacatalog
this following annotation; <http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
Also, added these following annotations to the ingress:
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
Fabio Grätz
05/31/2023, 8:10 PMseparateGrpcIngress
rings a bell, I am pretty sure I had to enable this.
<http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
is needed for flyteadmin (I don’t think for datacatalog since it is not exposed to the user?)
Are you sure it’s <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP
and not <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
? (source)Did you experience this thing? BTW, I don’t understand from your answer if you’re usingWe use both sporadically but have our own script based on flyte remote. However, in our current setup we cannot register … workflows with CLIs through the ingress because it uses IAP and we have not invested the time yet to try to make this work. Since all of our engineers work on VMs almost exclusively, we in the end gave flyteadmin a 2nd internal load balancer type service that is reachable from within the VPC.?flytectl/pyflyte
Ariel Kaspit
06/01/2023, 12:55 PMflytectl
and pyflyte
(unexpected HTTP status code received from server: 403 (Forbidden)
) came from Cloudflare. It was solved after enabling gRPC.Fabio Grätz
06/01/2023, 1:05 PMDavid Espejo (he/him)
06/01/2023, 2:48 PMAriel Kaspit
06/01/2023, 3:01 PMflyte
chart configuration it’s exactly as mentioned in values-gcp.yaml
. I enabled the separated ingress option:
common:
ingress:
host: "flyte.my.domain"
annotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
separateGrpcIngress: true
separateGrpcIngressAnnotations:
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
For the nginx-ingress-controller
helm chart configuration, please note I used a certificate from Cloudflare (not cert-manager) and I enabled gRPC networking in Cloudflare account (that was the reason for getting 403 forbidden
errors while accessing `flytectl`/`pyflyte`)
controller:
service:
annotations:
<http://external-dns.alpha.kubernetes.io/hostname|external-dns.alpha.kubernetes.io/hostname>: flyte.my.domain
config:
proxy-buffer-size: "16k"
extraArgs:
default-ssl-certificate: "namespace/secret-tls-name"
Fabio Grätz
06/01/2023, 3:46 PMDavid Espejo (he/him)
06/01/2023, 3:47 PM