Hey! I'm trying out flyte but I'm running into some issues with authentication. I've set up a new GKE cluster following https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual and then added Oauth2 authentication using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html. I can see that it is working on the browser

https://${MY_DOMAIN}/console

however I can't seem to get it to work on the command line, when running a command I get:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not ${MY_DOMAIN}"

My config looks something like (i've tried a few variations):

admin:
  endpoint: dns:///${MY_DOMAIN}
  authType: Pkce
  insecure: false
  clientId: ${MY_ID}
logger:
  show-source: true
  level: 0

Any hints on how to get around this?

Hey @Stephen McGroarty, welcome to the community! Can you please try this:

admin:
  endpoint: dns:///${MY_DOMAIN}
  authType: Pkce
  insecure: false
  clientId: ${MY_ID}
  insecureSkipVerify: true
logger:
  show-source: true
  level: 0

Ah thanks, hmm adding that gives me

Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken

Yeah it does actually, the guide says that can be ignored. Is that maybe not the case for CLI authentication?

does your console work?

Yeah it works and I'm using the google managed cert

Are you using the GKE ingress controller? Because there are issues due to health-checks with GRPC (which is used by the CLI). We had a few discussions about that in the deployment channel like but I don't think there's a good solution yet.

Yeah GKE ingress, using nginx

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx

Did you try the ingress setup from the manual guide https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ingress

If you want to use GKE ingress controller (which is required for Google managed certs), I think you to remove the nginx annotation.

Thanks! I'll give that a shot. Just to be clear what are the alternatives here? It says here

We use Google-managed SSL certificates.

https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ssl-certificate. I assume the

helm install cert-manager --namespace flyte --version v0.12.0 jetstack/cert-manager

is using the same google managed cert under the hood? Currently it my ingress config looks close to the default with:

<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"

      #<http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
      <http://networking.gke.io/managed-certificates|networking.gke.io/managed-certificates>: my-cert

    # --- separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx.
    separateGrpcIngress: true
    # --- Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled.
    separateGrpcIngressAnnotations:
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"

Are you saying I need to replace these

nginx.ingress

parts?

@Stephen McGroarty I think it might help to start with just the steps here: https://docs.flyte.org/en/latest/deployment/gcp/manual.html and NOT setup auth until we make sure everything is working properly…

Ok, thanks, I'll disable it

Cert-manager uses letsencrypt under the hood by default to issue certificates, not Google managed certs.

@Haytham Abuelfutuh with auth off it is able to write some of the demo code but oddly not all. It hits a gcp-storage issue. I am able to write directly on the terminal to that location fine

gsutil cp docs-requirements.txt gs://${MY_BUCKET}/metadata/admin/flytesnacks/development/core.flyte_basics.basic_workflow.my_wf/v1

@Stephen McGroarty It seems your

flyteadmin

pod does not have permissions to write to a bucket

Also instead of $MY_BUCKET variable can you use the actual bucket name in storage config for flytectl .the logs seem to indicate it using the variable and the not the value for it .