https://flyte.org logo
#ask-the-community
Title
# ask-the-community
s

Stephen McGroarty

03/22/2022, 4:11 AM
Hey! I'm trying out flyte but I'm running into some issues with authentication. I've set up a new GKE cluster following https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual and then added Oauth2 authentication using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html. I can see that it is working on the browser
https://${MY_DOMAIN}/console
however I can't seem to get it to work on the command line, when running a command I get:
Copy code
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not ${MY_DOMAIN}"
My config looks something like (i've tried a few variations):
Copy code
admin:
  endpoint: dns:///${MY_DOMAIN}
  authType: Pkce
  insecure: false
  clientId: ${MY_ID}
logger:
  show-source: true
  level: 0
Any hints on how to get around this?
h

Haytham Abuelfutuh

03/22/2022, 4:16 AM
Hey @Stephen McGroarty, welcome to the community! Can you please try this:
Copy code
admin:
  endpoint: dns:///${MY_DOMAIN}
  authType: Pkce
  insecure: false
  clientId: ${MY_ID}
  insecureSkipVerify: true
logger:
  show-source: true
  level: 0
when you go to
https://${MY_DOMAIN}/console
does the browser show any errors with the certificate?
s

Stephen McGroarty

03/22/2022, 5:12 AM
Ah thanks, hmm adding that gives me
Copy code
Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
Yeah it does actually, the guide says that can be ignored. Is that maybe not the case for CLI authentication?
k

Ketan (kumare3)

03/22/2022, 5:42 AM
does your console work?
Are these self signed certs?
s

Stephen McGroarty

03/22/2022, 5:48 AM
Yeah it works and I'm using the google managed cert
Ah I think I fixed part of the cert setup. The values.yaml defaults to
<http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
but I just used the ManagedCertificate directly (i.e needs to be `networking.gke.io/managed-certificates`)
Still have the above tho
s

Sören Brunk

03/22/2022, 7:08 AM
Are you using the GKE ingress controller? Because there are issues due to health-checks with GRPC (which is used by the CLI). We had a few discussions about that in the deployment channel like but I don't think there's a good solution yet.
s

Stephen McGroarty

03/22/2022, 7:41 AM
Yeah GKE ingress, using nginx
Copy code
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
p

Prafulla Mahindrakar

03/22/2022, 8:52 AM
Did you try the ingress setup from the manual guide https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ingress
s

Sören Brunk

03/22/2022, 1:21 PM
If you want to use GKE ingress controller (which is required for Google managed certs), I think you to remove the nginx annotation.
s

Stephen McGroarty

03/22/2022, 6:13 PM
Thanks! I'll give that a shot. Just to be clear what are the alternatives here? It says here
We use Google-managed SSL certificates.
https://docs.flyte.org/en/latest/deployment/gcp/manual.html#ssl-certificate. I assume the
helm install cert-manager --namespace flyte --version v0.12.0 jetstack/cert-manager
is using the same google managed cert under the hood? Currently it my ingress config looks close to the default with:
Copy code
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"

      #<http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
      <http://networking.gke.io/managed-certificates|networking.gke.io/managed-certificates>: my-cert

    # --- separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx.
    separateGrpcIngress: true
    # --- Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled.
    separateGrpcIngressAnnotations:
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
Are you saying I need to replace these
nginx.ingress
parts?
This really is a blank sheet cluster to experiment with flyte so my only requirement is authentication on the domain. Happy to change anything that lets me try out flyte in the cluster with auth
h

Haytham Abuelfutuh

03/22/2022, 6:19 PM
@Stephen McGroarty I think it might help to start with just the steps here: https://docs.flyte.org/en/latest/deployment/gcp/manual.html and NOT setup auth until we make sure everything is working properly…
just to avoid having too many moving parts
s

Stephen McGroarty

03/22/2022, 6:20 PM
Ok, thanks, I'll disable it
s

Sören Brunk

03/22/2022, 6:21 PM
Cert-manager uses letsencrypt under the hood by default to issue certificates, not Google managed certs.
s

Stephen McGroarty

03/22/2022, 6:49 PM
@Haytham Abuelfutuh with auth off it is able to write some of the demo code but oddly not all. It hits a gcp-storage issue. I am able to write directly on the terminal to that location fine
Copy code
gsutil cp docs-requirements.txt gs://${MY_BUCKET}/metadata/admin/flytesnacks/development/core.flyte_basics.basic_workflow.my_wf/v1
k

Ketan (kumare3)

03/22/2022, 8:33 PM
@Stephen McGroarty It seems your
flyteadmin
pod does not have permissions to write to a bucket
my_bucket
can you maybe either give permissions to the entire cluster or create a workload identity
p

Prafulla Mahindrakar

03/23/2022, 3:50 AM
Also instead of $MY_BUCKET variable can you use the actual bucket name in storage config for flytectl .the logs seem to indicate it using the variable and the not the value for it .
25 Views