Hey folks! A question:
I'm following the guide her...
# flyte-on-gcpa
alert-exabyte-15243
09/11/2024, 5:50 PMHey folks! A question:
I'm following the guide here: https://github.com/unionai-oss/deploy-flyte/blob/main/environments/gcp/flyte-core/README.md. I've followed the steps as stated; there is a DNS record configured and I can reach the console in browser. However, the grpc traffic doesn't seem to go through. The only suspicious thing I've found so far is the following: the certificate challenge is pending (for about 2 days now) and seems like the challenge is able to hit the ingress, however the acme-http-solver service sees errors like:
Copy code
"cert-manager/acmesolver: validating request" host="" path="*" base_path="." token="*"
"cert-manager/acmesolver: invalid base_path" host="" path="*" base_path="." token="*" expected_base_path="/.well-known/acme-challenge"alert-exabyte-15243
09/11/2024, 6:22 PMWhen i try to run
pyflyte Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2024-09-11T18:20:40.008462272+00:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused {grpc_status:14,
created_time:"2024-09-11T18:20:40.008460208+00:00"}]}" Copy code
endpoint: dns:///<our-domain>
insecure: true
insecureSkipVerify: truealert-exabyte-15243
09/11/2024, 7:17 PMMaybe another piece of useful info: we don't have any GCP Cloud DNS configuration configured; the A record points from a non-GCP provider to the Ingress. Console is available, however the certificate is not trusted by browser, which i guess is another indication of the certificate challenge issues seen from k8s side?
a
average-finland-92144
09/11/2024, 8:47 PMendpoint: dns:///<our-domain>
insecure: true
insecureSkipVerify: truecan you try changing this to
Copy code
endpoint: dns:///<our-domain>
insecure: false
insecureSkipVerify: truea
alert-exabyte-15243
09/11/2024, 8:50 PMChanged the config and ran pyflyte again, same result
failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refuseda
average-finland-92144
09/11/2024, 9:00 PMbut as far as I understand, DNS resolution is fine bc you get access to the console using the DNS name right?
Is that the full error message?
a
alert-exabyte-15243
09/11/2024, 9:05 PMYes, the console is accessible using the DNS name
The full error message:
a
average-finland-92144
09/11/2024, 9:14 PMI don't think this has to do with the certificate challenge.
Is this running behind an Ingress?
a
alert-exabyte-15243
09/11/2024, 9:23 PMas far as i understand, yes. The dns points to the ingress IP and, judging by the ingress configuration, http requests are being routed correctly
a
average-finland-92144
09/11/2024, 9:28 PMgot it, could you share the ingress configuration you're using? I just want to confirm you have all the necessary annotations
a
alert-exabyte-15243
09/12/2024, 10:07 AMSure! This is the
describe Copy code
Name: flyte-core │
│ Labels: <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm> │
│ Namespace: flyte │
│ Address: <IP-ADDRESS> │
│ Ingress Class: nginx │
│ Default backend: <default> │
│ TLS: │
│ flyte-secret-tls terminates <DOMAIN> │
│ Rules: │
│ Host Path Backends │
│ ---- ---- -------- │
│ <DOMAIN> │
│ /console flyteconsole:80 (172.16.0.21:8080) │
│ /console/* flyteconsole:80 (172.16.0.21:8080) │
│ /api flyteadmin:80 (172.16.0.22:8088) │
│ /api/* flyteadmin:80 (172.16.0.22:8088) │
│ /healthcheck flyteadmin:80 (172.16.0.22:8088) │
│ /v1/* flyteadmin:80 (172.16.0.22:8088) │
│ /.well-known flyteadmin:80 (172.16.0.22:8088) │
│ /.well-known/* flyteadmin:80 (172.16.0.22:8088) │
│ /login flyteadmin:80 (172.16.0.22:8088) │
│ /login/* flyteadmin:80 (172.16.0.22:8088) │
│ /logout flyteadmin:80 (172.16.0.22:8088) │
│ /logout/* flyteadmin:80 (172.16.0.22:8088) │
│ /callback flyteadmin:80 (172.16.0.22:8088) │
│ /callback/* flyteadmin:80 (172.16.0.22:8088) │
│ /me flyteadmin:80 (172.16.0.22:8088) │
│ /config flyteadmin:80 (172.16.0.22:8088) │
│ /config/* flyteadmin:80 (172.16.0.22:8088) │
│ /oauth2 flyteadmin:80 (172.16.0.22:8088) │
│ /oauth2/* flyteadmin:80 (172.16.0.22:8088) │
│ Annotations: <http://acme.cert-manager.io/http01-edit-in-place|acme.cert-manager.io/http01-edit-in-place>: true │
│ <http://cert-manager.io/issuer|cert-manager.io/issuer>: letsencrypt-production │
│ <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: / │
│ <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx │
│ <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-core │
│ <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte │
│ <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console │
│ <http://nginx.ingress.kubernetes.io/service-upstream|nginx.ingress.kubernetes.io/service-upstream>: true │
│ <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: true │
│ Events: <none>alert-exabyte-15243
09/12/2024, 10:07 AMGrpc ingress:
Copy code
Name: flyte-core-grpc │
│ Labels: app.kubernetes.io/managed-by=Helm │
│ Namespace: flyte │
│ Address: <IP> │
│ Ingress Class: nginx │
│ Default backend: <default> │
│ TLS: │
│ flyte-secret-tls terminates <DOMAIN> │
│ Rules: │
│ Host Path Backends │
│ ---- ---- -------- │
│ <DOMAIN> │
│ /.well-known/acme-challenge/<TOKEN> cm-acme-http-solver-6b4lg:8089 (172.16.0.27:8089) │
│ /flyteidl.service.SignalService flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.SignalService/* flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.AdminService flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.AdminService/* flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.DataProxyService flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.DataProxyService/* flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.AuthMetadataService flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.AuthMetadataService/* flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.IdentityService flyteadmin:81 (172.16.0.22:8089) │
│ /flyteidl.service.IdentityService/* flyteadmin:81 (172.16.0.22:8089) │
│ /grpc.health.v1.Health flyteadmin:81 (172.16.0.22:8089) │
│ /grpc.health.v1.Health/* flyteadmin:81 (172.16.0.22:8089) │
│ Annotations: acme.cert-manager.io/http01-edit-in-place: true │
│ cert-manager.io/issuer: letsencrypt-production │
│ ingress.kubernetes.io/rewrite-target: / │
│ kubernetes.io/ingress.class: nginx │
│ meta.helm.sh/release-name: flyte-core │
│ meta.helm.sh/release-namespace: flyte │
│ nginx.ingress.kubernetes.io/app-root: /console │
│ nginx.ingress.kubernetes.io/backend-protocol: GRPC │
│ nginx.ingress.kubernetes.io/service-upstream: true │
│ nginx.ingress.kubernetes.io/ssl-redirect: true │
│ Events: <none>alert-exabyte-15243
09/12/2024, 10:10 AMInterestingly, the http ingress doesn't explicitly specify the token route; but grpc one does (but i guess this might be grpc-specific?)
alert-exabyte-15243
09/12/2024, 7:07 PMAnother thing from nginx ingress logs, perhaps this could be useful as well:
Copy code
│ ingress-nginx-controller-86jhn 2024/09/12 19:05:37 [error] 304#304: *4162714 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 172.16.0.18, server: <DOMAIN>, request: "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1", upstream: "<grpc://192.168.0.136:8089>", host: "<DOMAIN>" │
│ ingress-nginx-controller-86jhn 172.16.0.18 - - [12/Sep/2024:19:05:37 +0000] "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1" 502 150 "-" "cert-manager-challenges/v1.13.2 (linux/amd64) cert-manager/432a489f5be77e3f4e2043564991a80e3bff6047" │
│ 277 0.001 [flyte-cm-acme-http-solver-6b4lg-8089] [] 192.168.0.136:8089 0 0.001 502 6bf36d69ceac8cc177f5b10fccca3a83a
average-finland-92144
09/13/2024, 6:20 PMoh that's interesting.
Is your setup running behind a proxy?
a
alert-exabyte-15243
09/13/2024, 9:09 PMWe have the IAP setup, but the network path for the flyte deployment doesn't seem to be behind it (at least I can access the console without going through the IAP). Not 100% sure if it has any effect on k8s cluster. Very happy to check/provide additional info, but will need pointers 🙂
a
average-finland-92144
09/16/2024, 3:32 PM@alert-exabyte-15243 what version of `flytekit`are you using?
a
alert-exabyte-15243
09/16/2024, 4:08 PM1.13.5a
average-finland-92144
09/16/2024, 4:18 PMok, AFAICT there's a different setup process if you want to use auth and the IAP, where you shouldn't be using the flyte-provided Ingress, for example. For the IAP, there's a plugin, so I was wondering if you have had the chance to follow the process described here:
https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-identity-aware-proxy#flytekit-identity-aware-proxy
a
alert-exabyte-15243
09/16/2024, 4:49 PMAhh, interesting! Thank you for the pointer; didn't see this page and it seems very relevant. Gonna take a look
Will let you know how things go!
54 Views