Dear all I ran into an inconsistency between flytectl and Fl

Question

Dear all I ran into an inconsistency between flytectl and FlyteRemote which has been reported <https flyte org slack com archives CP2HDHKE1 p1680811197406639|before> When trying to create a FlyteRemote with this config `admin ` `endpoint dns A B C D PPPPP` `insecure false` `authType Pkce` `insecureSkipVerify true` flytectl works just fine But when I try to fetch an execution on a FlyteRemote with this code `project = myproject ` `domain = development ` `execution = a5cphsfgc57nt6nxbknt ` `flyte config file = flyte config yaml ` `remote = FlyteRemote config=Config auto config file=flyte config file ` `flyte workflow execution = remote fetch execution project=project domain=domain name=execution ` I get the following error ```Traceback most recent call last File debug remote py line 12 in lt module gt flyte workflow execution = remote fetch execution project=project domain=domain name=execution File opt micromamba envs OHLI lib python3 8 site packages flytekit remote remote py line 353 in fetch execution self client get execution File opt micromamba envs OHLI lib python3 8 site packages flytekit clients friendly py line 582 in get execution super SynchronousFlyteClient self get execution File opt micromamba envs OHLI lib python3 8 site packages flytekit clients raw py line 43 in handler return fn args kwargs File opt micromamba envs OHLI lib python3 8 site packages flytekit clients raw py line 651 in get execution return self stub GetExecution get object request metadata=self metadata File opt micromamba envs OHLI lib python3 8 site packages grpc channel py line 1030 in call return end unary response blocking state call False None File opt micromamba envs OHLI lib python3 8 site packages grpc channel py line 910 in end unary response blocking raise InactiveRpcError state pytype disable=not instantiable grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses last error UNKNOWN ipv4 A B C D PPPPP Peer name A B C D is not in peer certificate debug error string = UNKNOWN failed to connect to all addresses last error UNKNOWN ipv4 A B C D PPPPP Peer name A B C D is not in peer certificate created time 2023 05 18T14 40 11 018710903+01 00 grpc status 14 gt ``` The cluster is running the flyte binary Helm chart in version 1 3 0 and I tried flytekit 1 2 11 1 3 0 and 1 6 1 all resulting in the same error message

David Espejo (he/him) · Answer

is `A B C D` added as a Subject Alt Name in your cert

Lukas Bommes · Answer

Thanks for the hint David I haven t edited the cert I thought that is not needed since I skip the certificate verification with`insecureSkipVerify true`But I will anyway try out adding the SAN and report back whether it works or not

Lukas Bommes · Answer

So I configured TLS for my Flyte endpoint which works all fine in the browser and with flytectl However `flytekit FlyteRemote` the debug script from my first post still fails Now with this error ``` asctime 2023 05 23 09 38 41 298 name flytekit levelname WARNING message FlyteSchema is deprecated use Structured Dataset instead WARNING root KeyRing not available tokens will not be cached Error No recommended backend was available Install a recommended 3rd party backend package or install the keyrings alt package if you want to use the non recommended backends See for details E0523 09 38 42 232413831 4238 Handshake failed with fatal error SSL ERROR SSL error 1000007d SSL routines OPENSSL internal CERTIFICATE VERIFY FAILED Traceback most recent call last File opt micromamba envs OHLI lib python3 8 site packages grpc interceptor py line 241 in continuation response call = self thunk new method with call File opt micromamba envs OHLI lib python3 8 site packages grpc interceptor py line 266 in with call return self with call request File opt micromamba envs OHLI lib python3 8 site packages grpc interceptor py line 257 in with call return call result call File opt micromamba envs OHLI lib python3 8 site packages grpc channel py line 343 in result raise self File opt micromamba envs OHLI lib python3 8 site packages grpc interceptor py line 241 in continuation response call = self thunk new method with call File opt micromamba envs OHLI lib python3 8 site packages grpc channel py line 957 in with call return end unary response blocking state call True None File opt micromamba envs OHLI lib python3 8 site packages grpc channel py line 849 in end unary response blocking raise InactiveRpcError state grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses last error UNKNOWN Ssl handshake failed debug error string = UNKNOWN Failed to pick subchannel created time 2023 05 23T09 38 42 234530152+01 00 children UNKNOWN failed to connect to all addresses last error UNKNOWN Ssl handshake failed created time 2023 05 23T09 38 42 234525804+01 00 grpc status 14 gt ``` I installed the CA inside the same container in which I execute the flyte script which throws this error And as mentioned flytectl work without problems from within the same container The flyte config looks now like this `admin ` `endpoint dns my domain lan 30657` `insecure false` `authType Pkce` Do you have any ideas Would be greatly appreciated slightly smiling face

Lukas Bommes · Answer

I fixed the problem by providing my root CA file as admin caCertFilePath in the flyte config However there is also a bug in flytekit clients auth helper Currently it is `credentials = grpc ssl channel credentials load cert cert file path ` However `load cert`returns an OpenSSL crypto X509 object But`grpc ssl channel credentialsexpects`a bytes string So I had to modify the call as follows to encode the X509 object as bytes `credentials = grpc ssl channel credentials crypto dump certificate crypto FILETYPE PEM load cert cert file path ` Can we fix this in main

Lukas Bommes · Answer

I created an issue <https github com flyteorg flyte issues 3715>

David Espejo (he/him) · Answer

Thanks < Lukas Bommes> for the issue the additional info and persevering through this An upcoming SIG auth should allow us to define goals around improving the auth experience in Flyte

Lukas Bommes · Answer

That s good news As of now I just monkeypatched the`load cert`function It works for me but it would be nice to have this permantely fixed

Enrico Rotundo · Answer

gt I fixed the problem by providing my root CA file as admin caCertFilePath in the flyte config hey < Lukas Bommes> where did you get the root CA file from I ve tried with the output of ```kubectl get cm kube root ca crt n flyte o jsonpath= data ca crt ``` but I still get `Peer name lt HOST gt is not in peer certificate` Also gt I configured TLS for my Flyte endpoint which works all fine in the browser and with flytectl What configuration did you use Something like ```tls hosts secretName example tls```

Enrico Rotundo · Answer

I ve enabled `common ingress tls` in values gcp yml fetched the related cert with `kubectl get secret flyte flyte tls n flyte o jsonpath= data tls crt | base64 decode gt tls crt` added the path of the cert in `admin caCertFilePath` in my local flyte config file Still running `pyflyte run remote` leads to a new error like the one you had `Handshake failed with fatal error SSL ERROR SSL error 1000007d SSL routines OPENSSL internal CERTIFICATE VERIFY FAILED `

David Espejo (he/him) · Answer

< Enrico Rotundo> what controller are you using for Ingress Also I don t think that K8s secret is the cert you need here It should a cert that a its Principal or SAN names includes the `host` name that points to your Ingress b Is signed by a trusted authority In such case you should be able to get the certificate chain root+ intermediates+ issued cert and instruct `flyteadmin` to use it by specifying it in the `config yaml` Otherwise it d be better for now to use `insecure true` and disable `insecureSkipVeryfy`

Enrico Rotundo · Answer

I ve just realized his so I ll try again with the latest instructions crossed fingers