So, I configured TLS for my Flyte endpoint which works all fine in the browser and with flytectl. However,

flytekit.FlyteRemote

(the debug script from my first post) still fails. Now, with this error

{"asctime": "2023-05-23 09:38:41,298", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
WARNING:root:KeyRing not available, tokens will not be cached. Error: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See <https://pypi.org/project/keyring> for details.
E0523 09:38:42.232413831    4238 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]> Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
Traceback (most recent call last):
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_interceptor.py", line 241, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_interceptor.py", line 266, in with_call
    return self._with_call(request,
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_interceptor.py", line 257, in _with_call
    return call.result(), call
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_channel.py", line 343, in result
    raise self
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_interceptor.py", line 241, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_channel.py", line 957, in with_call
    return _end_unary_response_blocking(state, call, True, None)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: Ssl handshake failed"
        debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-05-23T09:38:42.234530152+01:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Ssl handshake failed {created_time:"2023-05-23T09:38:42.234525804+01:00", grpc_status:14}]}"
>

I installed the CA inside the same container in which I execute the flyte script which throws this error. And as mentioned, flytectl work without problems from within the same container. The flyte config looks now like this

admin:

endpoint: dns:///my.domain.lan:30657

insecure: false

authType: Pkce

Do you have any ideas? Would be greatly appreciated. 🙂

I fixed the problem by providing my root CA file as admin.caCertFilePath in the flyte config. However, there is also a bug in flytekit.clients.auth_helper (https://github.com/flyteorg/flytekit/blob/master/flytekit/clients/auth_helper.py#L179). Currently, it is

credentials = grpc.ssl_channel_credentials(load_cert(<http://cfg.ca|cfg.ca>_cert_file_path))

However, `load_cert`returns an OpenSSL.crypto.X509 object. But`grpc.ssl_channel_credentialsexpects`a bytes string. So, I had to modify the call as follows (to encode the X509 object as bytes):

credentials = grpc.ssl_channel_credentials(crypto.dump_certificate(crypto.FILETYPE_PEM, load_cert(<http://cfg.ca|cfg.ca>_cert_file_path)))

Can we fix this in main?

I created an issue https://github.com/flyteorg/flyte/issues/3715

I’ve: • enabled

common.ingress.tls

in values-gcp.yml • fetched the related cert with

kubectl get secret flyte-flyte-tls -n flyte -o jsonpath="{['data']['tls\.crt']}" | base64 --decode > tls.crt

• added the path of the cert in

admin.caCertFilePath

in my local flyte config file Still running

pyflyte run --remote

leads to a new error like the one you had

Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.