cool-lifeguard-49380
08/18/2023, 5:04 PMcool-lifeguard-49380
08/18/2023, 5:08 PMflytescheduler precheck --config …/configs/*.yaml
. I’m seeing exactly the same behaviour though in flytepropeller.
I’m starting with this client config:
admin:
insecure: false
insecureSkipVerify: false
I’m getting:
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
This is expected, the certificate used by admin is self-signed as it is only used for the traffic between the load balancer and flyteadmin (or flytepropeller, … and flyteadmin).
I now change the client config to this:
insecure: false
insecureSkipVerify: true
I’m getting:
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: Client sent an HTTP request to an HTTPS server.
So we are ignoring the self-signed cert now, so far so good. The error Client sent an HTTP request to an HTTPS server.
is easy to fix, see below.
Instead of doing
insecure: false
insecureSkipVerify: true
I can also do:
insecure: false
insecureSkipVerify: false
caCertFilePath: …/cert.pem
And get the same error:
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: Client sent an HTTP request to an HTTPS server.
So at this point I have confidence that either setting insecureSkipVerify: true
or providing a caCertFilePath
works. The platform config is correctly used here (verified with debugger), and we can either ignore the self-signed cert, or we validate it with the provided cert.pem
. Good.
Let’s solve the Response: Client sent an HTTP request to an HTTPS server.
error next.
insecure: false # <- Apparently not enough
insecureSkipVerify: false
tokenUrl: <https://localhost:8080/oauth2/token> # Explicitly telling the client to use TLS when requesting token
caCertFilePath: …/cert.pem
Now I get this error message:
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://localhost:8080/oauth2/token>": tls: failed to verify certificate: x509: certificate signed by unknown authority
(While about the certificate authority again, it’s slightly different from the one above.)
The error originates here where we try to get a token but seem not to honour the client config which would tell us to either ignore the self-signed cert or gives us a cert.pem
to verify it.
I understand running admin with TLS is not used much but I’d like to fix this if possible.
Tagging a bunch of people I saw modified the mentioned code passages, hoping that somebody can give me a pointer how to proceed here 🙈
@icy-agent-73298 @thankful-minister-83577 @acceptable-policeman-57188 @white-painting-22485@high-park-82026cool-lifeguard-49380
08/24/2023, 5:40 PM