elegant-toddler-67101
05/31/2023, 4:41 PMseparateGrpcIngress
? I understand from the documentation that it is Required for certain ingress controllers like nginx.
2. After deploying nginx (which uses our own tls certificate, not self-signed with cert-manager), I’m getting 502 bad gateway errors accessing the console (flyte.my.domain)… In the flyteadmin
logs I see this authentication error: Failed to refresh tokens. Restarting login flow. Error: [TOKEN_REFRESH_FAILURE] Error refreshing token, caused by: oauth2: cannot fetch token: 400 Bad Request
(it worth mentioning I don’t see any errors in Okta, which we configured there the authorization server)
3. Regarding accessing within the CLI, while trying to use flytectl
I’m still getting the same authentication error: PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
- Did you experience this thing? BTW, I don’t understand from your answer if you’re using flytectl/pyflyte
?
Thanks in advanced! And sorry for all the questions, I appreciate your help!elegant-toddler-67101
05/31/2023, 4:42 PMpyflyte
and the error is:
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Received http2 header with status: 403"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Received http2 header with status: 403", grpc_status:7, created_time:"2023-05-31T19:41:03.999452+03:00"}"
It seems to still have issues with the gRPC service… Did I miss something?elegant-toddler-67101
05/31/2023, 4:44 PMflyteadmin
and datacatalog
this following annotation; <http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
Also, added these following annotations to the ingress:
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
cool-lifeguard-49380
05/31/2023, 8:10 PMseparateGrpcIngress
rings a bell, I am pretty sure I had to enable this.
<http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
is needed for flyteadmin (I don’t think for datacatalog since it is not exposed to the user?)
Are you sure it’s <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP
and not <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
? (source)cool-lifeguard-49380
05/31/2023, 8:11 PMcool-lifeguard-49380
05/31/2023, 8:11 PMcool-lifeguard-49380
05/31/2023, 8:13 PMDid you experience this thing? BTW, I don’t understand from your answer if you’re usingWe use both sporadically but have our own script based on flyte remote. However, in our current setup we cannot register … workflows with CLIs through the ingress because it uses IAP and we have not invested the time yet to try to make this work. Since all of our engineers work on VMs almost exclusively, we in the end gave flyteadmin a 2nd internal load balancer type service that is reachable from within the VPC.?flytectl/pyflyte
elegant-toddler-67101
06/01/2023, 12:55 PMelegant-toddler-67101
06/01/2023, 12:58 PMflytectl
and pyflyte
(unexpected HTTP status code received from server: 403 (Forbidden)
) came from Cloudflare. It was solved after enabling gRPC.cool-lifeguard-49380
06/01/2023, 1:05 PMcool-lifeguard-49380
06/01/2023, 1:06 PMcool-lifeguard-49380
06/01/2023, 1:06 PMaverage-finland-92144
06/01/2023, 2:48 PMelegant-toddler-67101
06/01/2023, 3:01 PMflyte
chart configuration it’s exactly as mentioned in values-gcp.yaml
. I enabled the separated ingress option:
common:
ingress:
host: "flyte.my.domain"
annotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
separateGrpcIngress: true
separateGrpcIngressAnnotations:
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
For the nginx-ingress-controller
helm chart configuration, please note I used a certificate from Cloudflare (not cert-manager) and I enabled gRPC networking in Cloudflare account (that was the reason for getting 403 forbidden
errors while accessing `flytectl`/`pyflyte`)
controller:
service:
annotations:
<http://external-dns.alpha.kubernetes.io/hostname|external-dns.alpha.kubernetes.io/hostname>: flyte.my.domain
config:
proxy-buffer-size: "16k"
extraArgs:
default-ssl-certificate: "namespace/secret-tls-name"
cool-lifeguard-49380
06/01/2023, 3:46 PMcool-lifeguard-49380
06/01/2023, 3:46 PMaverage-finland-92144
06/01/2023, 3:47 PMaverage-finland-92144
08/14/2023, 9:42 PM