https://flyte.org logo
#contribute
Title
# contribute
f

Fabio Grätz

08/18/2023, 5:04 PM
Hey everyone 🙂 I could use some help with understanding the admin client in flyteidl, I’m struggling to get it to work when admin uses TLS. Details in 🧵
I need to run flyteadmin with TLS to get it to work with a GCE ingress. Needed for this: https://github.com/flyteorg/flyte/issues/3965 Flyteadmin already supports being given a cert and starting a TLS server. I’m struggling to configure the admin client for flytescheduler and flytepropeller to talk to an admin with TLS. I’m debugging the init container by flytescheduler which runs
flytescheduler precheck --config …/configs/*.yaml
. I’m seeing exactly the same behaviour though in flytepropeller. I’m starting with this client config:
Copy code
admin:
 insecure: false
 insecureSkipVerify: false
I’m getting:
Copy code
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
This is expected, the certificate used by admin is self-signed as it is only used for the traffic between the load balancer and flyteadmin (or flytepropeller, … and flyteadmin). I now change the client config to this:
Copy code
insecure: false
 insecureSkipVerify: true
I’m getting:
Copy code
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: Client sent an HTTP request to an HTTPS server.
So we are ignoring the self-signed cert now, so far so good. The error
Client sent an HTTP request to an HTTPS server.
is easy to fix, see below. Instead of doing
Copy code
insecure: false
 insecureSkipVerify: true
I can also do:
Copy code
insecure: false
 insecureSkipVerify: false
 caCertFilePath: …/cert.pem
And get the same error:
Copy code
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: Client sent an HTTP request to an HTTPS server.
So at this point I have confidence that either setting
insecureSkipVerify: true
or providing a
caCertFilePath
works. The platform config is correctly used here (verified with debugger), and we can either ignore the self-signed cert, or we validate it with the provided
cert.pem
. Good. Let’s solve the
Response: Client sent an HTTP request to an HTTPS server.
error next.
Copy code
insecure: false # <- Apparently not enough
 insecureSkipVerify: false
 tokenUrl: <https://localhost:8080/oauth2/token> # Explicitly telling the client to use TLS when requesting token
 caCertFilePath: …/cert.pem
Now I get this error message:
Copy code
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://localhost:8080/oauth2/token>": tls: failed to verify certificate: x509: certificate signed by unknown authority
(While about the certificate authority again, it’s slightly different from the one above.) The error originates here where we try to get a token but seem not to honour the client config which would tell us to either ignore the self-signed cert or gives us a
cert.pem
to verify it. I understand running admin with TLS is not used much but I’d like to fix this if possible. Tagging a bunch of people I saw modified the mentioned code passages, hoping that somebody can give me a pointer how to proceed here 🙈 @Prafulla Mahindrakar @Yee @katrina @Andrew Dye@Haytham Abuelfutuh
While the issue persists, I now worked around it in the effort to create a way to deploy flyte with google identity aware proxy :) https://github.com/flyteorg/flytekit/blob/c8060d35a56a7c5bf4cd1bc71bae4011afd3c9f2/plugins/flytekit-identity-aware-proxy/README.md So help is not necessarily needed anymore, the setup is actually rather elegant this way. Also automatic reloading of the certificate is solved like this ..
58 Views