Thread
#ask-the-community
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    Hey! I'm trying out flyte but I'm running into some issues with authentication. I've set up a new GKE cluster following https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual and then added Oauth2 authentication using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html. I can see that it is working on the browser
    https://${MY_DOMAIN}/console
    however I can't seem to get it to work on the command line, when running a command I get:
    Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not ${MY_DOMAIN}"
    My config looks something like (i've tried a few variations):
    admin:
      endpoint: dns:///${MY_DOMAIN}
      authType: Pkce
      insecure: false
      clientId: ${MY_ID}
    logger:
      show-source: true
      level: 0
    Any hints on how to get around this?
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    6 months ago
    Hey @Stephen McGroarty, welcome to the community! Can you please try this:
    admin:
      endpoint: dns:///${MY_DOMAIN}
      authType: Pkce
      insecure: false
      clientId: ${MY_ID}
      insecureSkipVerify: true
    logger:
      show-source: true
      level: 0
    when you go to
    https://${MY_DOMAIN}/console
    does the browser show any errors with the certificate?
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    Ah thanks, hmm adding that gives me
    Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
    Yeah it does actually, the guide says that can be ignored. Is that maybe not the case for CLI authentication?
    Ketan (kumare3)

    Ketan (kumare3)

    6 months ago
    does your console work?
    Are these self signed certs?
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    Yeah it works and I'm using the google managed cert
    Ah I think I fixed part of the cert setup. The values.yaml defaults to
    <http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
    but I just used the ManagedCertificate directly (i.e needs to be `networking.gke.io/managed-certificates`)
    Still have the above tho
    s

    Sören Brunk

    6 months ago
    Are you using the GKE ingress controller? Because there are issues due to health-checks with GRPC (which is used by the CLI). We had a few discussions about that in the deployment channel like but I don't think there's a good solution yet.
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    Yeah GKE ingress, using nginx
    apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
    kind: Ingress
    metadata:
      annotations:
        <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    p

    Prafulla Mahindrakar

    6 months ago
    Did you try the ingress setup from the manual guide https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ingress
    s

    Sören Brunk

    6 months ago
    If you want to use GKE ingress controller (which is required for Google managed certs), I think you to remove the nginx annotation.
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    Thanks! I'll give that a shot. Just to be clear what are the alternatives here? It says here
    We use Google-managed SSL certificates.
    https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ssl-certificate. I assume the
    helm install cert-manager --namespace flyte --version v0.12.0 jetstack/cert-manager
    is using the same google managed cert under the hood? Currently it my ingress config looks close to the default with:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
          <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    
          #<http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
          <http://networking.gke.io/managed-certificates|networking.gke.io/managed-certificates>: my-cert
    
        # --- separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx.
        separateGrpcIngress: true
        # --- Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled.
        separateGrpcIngressAnnotations:
          <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
    Are you saying I need to replace these
    nginx.ingress
    parts?
    This really is a blank sheet cluster to experiment with flyte so my only requirement is authentication on the domain. Happy to change anything that lets me try out flyte in the cluster with auth
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    6 months ago
    @Stephen McGroarty I think it might help to start with just the steps here: https://docs.flyte.org/en/latest/deployment/gcp/manual.html and NOT setup auth until we make sure everything is working properly…
    just to avoid having too many moving parts
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    Ok, thanks, I'll disable it
    s

    Sören Brunk

    6 months ago
    Cert-manager uses letsencrypt under the hood by default to issue certificates, not Google managed certs.
    Stephen McGroarty

    Stephen McGroarty

    6 months ago
    @Haytham Abuelfutuh with auth off it is able to write some of the demo code but oddly not all. It hits a gcp-storage issue. I am able to write directly on the terminal to that location fine
    gsutil cp docs-requirements.txt gs://${MY_BUCKET}/metadata/admin/flytesnacks/development/core.flyte_basics.basic_workflow.my_wf/v1
    Ketan (kumare3)

    Ketan (kumare3)

    6 months ago
    @Stephen McGroarty It seems your
    flyteadmin
    pod does not have permissions to write to a bucket
    my_bucket
    can you maybe either give permissions to the entire cluster or create a workload identity
    p

    Prafulla Mahindrakar

    6 months ago
    Also instead of $MY_BUCKET variable can you use the actual bucket name in storage config for flytectl .the logs seem to indicate it using the variable and the not the value for it .