Does anyone have experience with TLS SSL between the Ingress

Question

Does anyone have experience with TLS SSL between the Ingress and Flyte lock lock I m currently trying to get a reference implementation to work for deploying Flyte on GCP with the GKE Ingress controller instead of nginx wich would allow us to use for authentication IAP requires the user to login at the load balancer before a request reaches the backend instead of being redirected to the login screen by the backend Because this is more secure it s a requirement by our company and actually also my previous one to use IAP To make gRPC backends work with the GKE Ingress HTTP2 needs to be enabled for the backend Flyte already does so Using HTTP2 between the load balancer and the backend requires TLS between lb and backend So I m trying to deploy Flyte with this config ```configmap adminServer server httpPort 8088 grpcPort 8089 security secure false true ssl certificateFile etc tls cert pem keyFile etc tls key pem ``` Certificate is self signed and mounted into the flyteadmin pod via a secret I can t talk to flyteadmin though and would appreciate help getting this to work slightly smiling face Details in thread

Fabio Grätz · Answer

I got a minimal working example hello world gRPC app to work with GKE Ingress and IAP For it to work the server needs to use a TLS certificate which can be self signed I create the certificate with `openssl req x509 newkey rsa 4096 keyout key pem out cert pem days 365 nodes subj CN=0 0 0 0 ` which works for my hello world example But from another pod in the flyte namespace I cannot talk to admin apart from health check ```curl v insecure SSLv3 TLS handshake Client hello 1 SSLv3 TLS handshake Server hello 2 SSLv3 TLS handshake CERT 11 SSLv3 TLS handshake Server key exchange 12 SSLv3 TLS handshake Server finished 14 SSLv3 TLS handshake Client key exchange 16 SSLv3 TLS change cipher Client hello 1 SSLv3 TLS handshake Finished 20 SSLv3 TLS change cipher Client hello 1 SSLv3 TLS handshake Finished 20 gt GET healthcheck HTTP 1 1 gt User Agent curl 7 35 0 gt Host flyteadmin gt Accept gt lt HTTP 1 1 200 OK lt Date Fri 30 Jun 2023 17 53 27 GMT lt Content Length 0``` But ```curl v insecure SSLv3 TLS handshake Client hello 1 SSLv3 TLS handshake Server hello 2 SSLv3 TLS handshake CERT 11 SSLv3 TLS handshake Server key exchange 12 SSLv3 TLS handshake Server finished 14 SSLv3 TLS handshake Client key exchange 16 SSLv3 TLS change cipher Client hello 1 SSLv3 TLS handshake Finished 20 SSLv3 TLS change cipher Client hello 1 SSLv3 TLS handshake Finished 20 gt GET api v1 projects HTTP 1 1 gt User Agent curl 7 35 0 gt Host flyteadmin gt Accept gt lt HTTP 1 1 503 Service Unavailable lt Content Type application json lt Date Fri 30 Jun 2023 17 53 45 GMT lt Content Length 337 lt error connection error desc = transport authentication handshake failed x509 certificate is not valid for any names but wanted to match 0 0 0 0 8088 code 14 message connection error desc = transport authentication handshake failed x509 certificate is not valid for any names but wanted to match 0 0 0 0 8088 ``` Does anyone know how the certificate needs to be created

Fabio Grätz · Answer

Screenshot 2023 06 30 at 19 54 57 png

Fabio Grätz · Answer

Update With `openssl req x509 newkey rsa 4096 keyout key pem out cert pem days 365 nodes subj CN=0 0 0 0 addext subjectAltName = DNS 0 0 0 0 8088 ` I can now to to the http server of admin from within the cluster and also flyte console works

Fabio Grätz · Answer

```curl insecure <https flyteadmin 80 api v1 projects> projects id flytesnacks name flytesnacks domains id development name development id staging name staging id production name production description flytesnacks description ```

Fabio Grätz · Answer

But I still cannot make requests to the gRPC server confused `flytectl get projects` works when disabling ssl between lb and admin

Fabio Grätz · Answer

Flyte scheduler init container also fails with this error ```panic rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 10 88 13 168 81 connect connection refused ``` This is the flyteadmin IP

Fabio Grätz · Answer

Screenshot 2023 06 30 at 20 39 47 png

Fabio Grätz · Answer

So something is still off with the certificate I assume Would really appreciate if somebody knows how the certificate needs to be generated see no evil

jeev · Answer

< Fabio Grätz> it doesnt work within the cluster either ```flytectl get projects admin endpoint=flyteadmin 81 admin insecureSkipVerify=true```

Fabio Grätz · Answer

I actually didn t try that but will now hourglass flowing sand

Fabio Grätz · Answer

``` bin flytectl get projects admin endpoint=flyteadmin 81 admin insecureSkipVerify=true INFO 0000 0 Couldn t find a config file Relying on env vars and pflags json level warning msg using insecureSkipVerify Server s certificate chain and host name wont be verified Caution shouldn t be used for production usecases ts 2023 06 30T18 54 04Z Error Connection Info Endpoint flyteadmin 81 InsecureConnection false AuthMode ClientSecret rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 10 88 13 201 81 connect connection refused json level error msg Connection Info Endpoint flyteadmin 81 InsecureConnection false AuthMode ClientSecret rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 10 88 13 201 81 connect connection refused ts 2023 06 30T18 54 04Z ``` Not it doesn t also from within the cluster Actually this makes me very happy see no evil if this had worked but the load balancer hadn t I would have become really pessimistic as the lb is very painful to debug

Fabio Grätz · Answer

As a sanity check From the same pod this works ```curl insecure <https flyteadmin 80 api v1 projects> projects id flytesnacks name flytesnacks domains id development name development id staging name staging id production name production description flytesnacks description ```

jeev · Answer

so the grpc server is basically not starting it looks like

jeev · Answer

anything in admin logs

Fabio Grätz · Answer

Only this ```k logs f flyteadmin 5764f7c76c wthhs Defaulted container flyteadmin out of flyteadmin run migrations init seed projects init generate secrets init time= 2023 06 30T18 49 07Z level=info msg= Using config file etc flyte config clusters yaml etc flyte config db yaml etc flyte config domain yaml etc flyte config namespace config yaml etc flyte config remoteData yaml etc flyte config server yaml etc flyte config storage yaml etc flyte config task resource defaults yaml ```

jeev · Answer

maybe crank up the log level to 5

Fabio Grätz · Answer

Yes will do Need to figure out where in the config i need to set this hourglass flowing sand

jeev · Answer

are you using flyte core

jeev · Answer

```configmap logger logger show source true level 5```

jeev · Answer

in the values

Fabio Grätz · Answer

Restarting now

Fabio Grätz · Answer

Nothing in the logs thinking face ```k logs f flyteadmin df5754849 jwjqn Defaulted container flyteadmin out of flyteadmin run migrations init seed projects init generate secrets init time= 2023 06 30T19 05 36Z level=info msg= Using config file etc flyte config clusters yaml etc flyte config db yaml etc flyte config domain yaml etc flyte config logger yaml etc flyte config namespace config yaml etc flyte config remoteData yaml etc flyte config server yaml etc flyte config storage yaml etc flyte config task resource defaults yaml json src viper go 398 level debug msg Config section auth updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section notifications updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section cloudevents updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section task resources updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section task type whitelist updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 400 level debug msg Config section admin updated Firing updated event ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section propeller updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section server updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section scheduler updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section externalevents updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section cluster resources updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section queues updated No update handler registered ts 2023 06 30T19 05 36Z json src viper go 398 level debug msg Config section namespace mapping updated No update handler registered ts 2023 06 30T19 05 36Z json src service go 68 level info msg setting metrics keys to project domain wf task phase tasktype runtime type runtime version app name ts 2023 06 30T19 05 36Z json src cert utils go 23 level info msg Constructing SSL credentials ts 2023 06 30T19 05 36Z json src service go 79 level info msg Registering default middleware with blanket auth validation ts 2023 06 30T19 05 36Z json src service go 94 level info msg Creating gRPC server without authentication ts 2023 06 30T19 05 36Z json src server go 97 level info msg Starting profiling server on port 10254 ts 2023 06 30T19 05 36Z json src database go 171 level info msg Set connection pool values to MaxOpenConnections 100 OpenConnections 1 InUse 0 Idle 1 WaitCount 0 WaitDuration 0s MaxIdleClosed 0 MaxIdleTimeClosed 0 MaxLifetimeClosed 0 ts 2023 06 30T19 05 37Z json src client go 73 level debug msg successfully loaded kube configuration from in cluster config ts 2023 06 30T19 05 37Z json src base go 98 level info msg Successfully created a workflow executor engine ts 2023 06 30T19 05 37Z json src factory go 177 level info msg Using default noop notifications publisher implementation for config type local ts 2023 06 30T19 05 37Z json src factory go 126 level info msg Using default noop notifications processor implementation for config type local ts 2023 06 30T19 05 37Z json src base go 106 level info msg Started processing notifications ts 2023 06 30T19 05 37Z json src noop notifications go 43 level debug msg call to noop start processing ts 2023 06 30T19 05 37Z json src factory go 86 level info msg Using default flyte scheduler implementation ts 2023 06 30T19 05 37Z json src factory go 104 level info msg Using default noop workflow executor implementation for cloud provider type local ts 2023 06 30T19 05 37Z json src factory go 88 level info msg Using default noop remote url implementation for cloud provider type gcs ts 2023 06 30T19 05 37Z json src base go 150 level info msg Successfully initialized a new scheduled workflow executor ts 2023 06 30T19 05 37Z json src base go 152 level info msg Starting the scheduled workflow executor ts 2023 06 30T19 05 37Z json src base go 166 level info msg Initializing a new AdminService ts 2023 06 30T19 05 37Z json src database go 171 level info msg Set connection pool values to MaxOpenConnections 100 OpenConnections 1 InUse 0 Idle 1 WaitCount 0 WaitDuration 0s MaxIdleClosed 0 MaxIdleTimeClosed 0 MaxLifetimeClosed 0 ts 2023 06 30T19 05 37Z json src signal service go 77 level info msg Initializing a new SignalService ts 2023 06 30T19 05 37Z 2023 06 30 19 06 31 go pkg mod gorm io gorm v1 24 1 0 20221019064659 5dd2bb482755 callbacks go 134 5 209ms rows 1 SELECT FROM projects WHERE state = 1 ORDER BY identifier asc 2023 06 30 19 06 47 go pkg mod gorm io gorm v1 24 1 0 20221019064659 5dd2bb482755 callbacks go 134 3 220ms rows 1 SELECT FROM projects WHERE state = 1 ORDER BY identifier asc```

Fabio Grätz · Answer

The 2 queries in the end both come from curl gt http server

jeev · Answer

the last time i tried doing this i dont think i used a cert at all for internal

jeev · Answer

let me confirm

Fabio Grätz · Answer

Screenshot 2023 06 22 at 18 15 17 png

Fabio Grätz · Answer

I m pretty sure that on GCP there is no way without TLS between LB and backend

Fabio Grätz · Answer

Screenshot 2023 06 22 at 18 44 48 png

jeev · Answer

see here <https github com flyteorg flyte pull 3513 files>

Fabio Grätz · Answer

Yes with this ingress controller this might be different

jeev · Answer

same issue

jeev · Answer

didnt work without insecure=false

jeev · Answer

did you try without any changes to flyteadmin config

jeev · Answer

just to rule it out

Fabio Grätz · Answer

You mean turn of ssl again and see if it works then

jeev · Answer

im not doing anything different for ALB either

jeev · Answer

turn off SSL in the flyteadmin config but inform the LB to hit the backend using http2

jeev · Answer

i think youdo that via the port spec scheme

Fabio Grätz · Answer

Fabio Grätz · Answer

Here it says explicitly that for the GKE Ingress controller this doesn t work without TLS

Fabio Grätz · Answer

With my minimal working example greeter gRPC app I confirmed that the lb only forwards requests to the backend once a cert is added to the backend

Fabio Grätz · Answer

But I m happy to try again ^^

Fabio Grätz · Answer

Thanks for supporting btw pray pray

jeev · Answer

see here <https cloud google com kubernetes engine docs how to ingress http2>

Fabio Grätz · Answer

On this page it says gt Note To ensure the load balancer can make a correct HTTP2 request to your backend your backend must be configured with SSL For more information on what types of certificates are accepted see <https cloud google com load balancing docs ssl certificates backend encryption|Encryption from the load balancer to the backends>

jeev · Answer

hmm maybe that app auto generates certs then

Fabio Grätz · Answer

The one in the docs does yes I checked this

Fabio Grätz · Answer

You can see it in the logs

Fabio Grätz · Answer

With `insecure=true` and without `ssl` in the helm values it works ``` bin flytectl get projects admin endpoint=flyteadmin 81 admin insecure=true INFO 0000 0 Couldn t find a config file Relying on env vars and pflags | ID | NAME | DESCRIPTION | | flytesnacks | flytesnacks | flytesnacks description | ``` The load balancer won t work but this shows that the rest of the config is likely fine

jeev · Answer

+1

jeev · Answer

```configmap adminServer server httpPort 8088 grpcPort 8089 security secure true ssl certificateFile etc tls cert pem keyFile etc tls key pem ```

jeev · Answer

is this what you are running

jeev · Answer

if you just set `secure false` in cluster connections work

Fabio Grätz · Answer

``` server httpPort 8088 grpcPort 8089 security secure true ssl certificateFile etc tls cert pem keyFile etc tls key pem ```

Fabio Grätz · Answer

If I set `secure false` and comment out `ssl` in cluster works I assume it would also if I didn t comment out ssl since it shouldn t have an effect

jeev · Answer

right

jeev · Answer

ok

jeev · Answer

ill have to play around with this

jeev · Answer

i suspect we can repro this in sandbox for faster iteration

jeev · Answer

can confirm that adding enabling ssl breaks flyteadmin grpc on sandbox as well

jeev · Answer

not sure whats happening though

Haytham Amin · Answer

< Drew OConnor>

Neil S · Answer

Has there been any movement on this issue I m seeing the same thing on an OpenShift deployment where after enabling secure and ssl entries using passthrough for tls on the OpenShift Route I receive the effectively same issue of cert errors with 0 0 0 0 8088 not being a valid SAN in the logs but I am unable to generate by policy a DNS SAN with 0 0 0 0 8088 and an IP SAN of 0 0 0 0 is ineffectual so I m even further back on this issue