@Ariel Kaspit if you use the GKE ingress, I guess you configured the routing rules for flyteadmin and flyteconsole there. Is the flytedamin backend shown as healthy for you in the GKE ingress?

Yes, all services in use by the ingress are healthy. But I did a little workaround with this thing, I configured the healthchecks for the backendconfig which in use by the service

Ok but you agree that if one doesn’t customize this, out of the box the GKE ingress won’t work? (So I don’t write something that is out of date into the issue)

Totally correct

Ok cool, then I will go ahead and create the issue.

Yes, I’ll comment with everything. BTW, I checked and the backends for the flyteadmin service on port 81 (which is the gRPC) is not healthy since (exactly what you said) GKE ingress controller cannot handle gRPC readiness and liveness probes of flyteadmin, for this one I didn’t find anything. I tried to create a separated backendconfigs for HTTP and gRPC requests…

apiVersion: <http://cloud.google.com/v1|cloud.google.com/v1>
kind: BackendConfig
metadata:
  name: flyteadmin-backendconfig-grpc
  namespace: flyte
spec:
  securityPolicy:
    name: allow-only-cloudflare-source-ips
  healthCheck:
    requestPath: /grpc.health.v1.Health/Check
    port: 8089
    type: HTTP2
---
apiVersion: <http://cloud.google.com/v1|cloud.google.com/v1>
kind: BackendConfig
metadata:
  name: flyteadmin-backendconfig-http
  namespace: flyte
spec:
  securityPolicy:
    name: allow-only-cloudflare-source-ips
  healthCheck:
    requestPath: /healthcheck
    port: 8088

I tried lots of different paths, and also tried this path

/grpc.health.v1.Health/Check

since I found this article which explains how to enable healthchecks for gRPC on Kubernetes. But it requires an extra configuration. Anyway, it wasn’t specified in the documentation so it took me an extra time unfortunately.