I need to run flyteadmin with TLS to get it to work with a GCE ingress. Needed for this: https://github.com/flyteorg/flyte/issues/3965 Flyteadmin already supports being given a cert and starting a TLS server. I’m struggling to configure the admin client for flytescheduler and flytepropeller to talk to an admin with TLS. I’m debugging the init container by flytescheduler which runs

flytescheduler precheck --config …/configs/*.yaml

. I’m seeing exactly the same behaviour though in flytepropeller. I’m starting with this client config:

admin:
 insecure: false
 insecureSkipVerify: false

I’m getting:

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

This is expected, the certificate used by admin is self-signed as it is only used for the traffic between the load balancer and flyteadmin (or flytepropeller, … and flyteadmin). I now change the client config to this:

insecure: false
 insecureSkipVerify: true

I’m getting:

rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: Client sent an HTTP request to an HTTPS server.

So we are ignoring the self-signed cert now, so far so good. The error

Client sent an HTTP request to an HTTPS server.

is easy to fix, see below. Instead of doing

insecure: false
 insecureSkipVerify: true

I can also do:

insecure: false
 insecureSkipVerify: false
 caCertFilePath: …/cert.pem

And get the same error:

rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: Client sent an HTTP request to an HTTPS server.

So at this point I have confidence that either setting

insecureSkipVerify: true

or providing a

caCertFilePath

works. The platform config is correctly used here (verified with debugger), and we can either ignore the self-signed cert, or we validate it with the provided

cert.pem

. Good. Let’s solve the

Response: Client sent an HTTP request to an HTTPS server.

error next.

insecure: false # <- Apparently not enough
 insecureSkipVerify: false
 tokenUrl: <https://localhost:8080/oauth2/token> # Explicitly telling the client to use TLS when requesting token
 caCertFilePath: …/cert.pem

Now I get this error message:

rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://localhost:8080/oauth2/token>": tls: failed to verify certificate: x509: certificate signed by unknown authority

(While about the certificate authority again, it’s slightly different from the one above.) The error originates here where we try to get a token but seem not to honour the client config which would tell us to either ignore the self-signed cert or gives us a

cert.pem

to verify it. I understand running admin with TLS is not used much but I’d like to fix this if possible. Tagging a bunch of people I saw modified the mentioned code passages, hoping that somebody can give me a pointer how to proceed here 🙈 @Prafulla Mahindrakar @Yee @katrina @Andrew Dye@Haytham Abuelfutuh

While the issue persists, I now worked around it in the effort to create a way to deploy flyte with google identity aware proxy :) https://github.com/flyteorg/flytekit/blob/c8060d35a56a7c5bf4cd1bc71bae4011afd3c9f2/plugins/flytekit-identity-aware-proxy/README.md So help is not necessarily needed anymore, the setup is actually rather elegant this way. Also automatic reloading of the certificate is solved like this ..