Trying to move our deployment beyond port forwarding. I followed the instructions here and got everything working correctly for the console/UI with TLS, but I’m having difficulty with

flytectl

. When I run

flytectl -n flytesnacks -p development get tasks

I get

Error: Connection Info: [Endpoint: dns:///<redacted>, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type
{"json":{},"level":"error","msg":"Connection Info: [Endpoint: dns:///<redacted>, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type","ts":"2023-12-20T10:48:53-08:00"}

When I try curl on the

grpc.health.v1.Health

endpoint

*   Trying <redacted>...
* Connected to <redacted> (<redacted>) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=<redacted>
*  start date: Dec 20 14:30:05 2023 GMT
*  expire date: Mar 19 14:30:04 2024 GMT
*  subjectAltName: host "<redacted>" matched cert's "<redacted>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: <redacted>]
* h2 [:path: /grpc.health.v1.Health]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x11d814200)
> GET /grpc.health.v1.Health HTTP/2
> Host: <redacted>
> User-Agent: curl/8.1.2
> Accept: */*
>
* Unsupported response code in HTTP response
* Connection #0 to host <redacted> left intact

When I check the ingress-nginx controller’s logs I see

2023/12/20 18:57:56 [error] 241#241: *3348260 recv() failed (104: Connection reset by peer) while reading upstream, client: 10.244.1.1, server: <redacted>, request: "GET /grpc.health.v1.Health HTTP/2.0", upstream: "<http://10.244.1.83:8089/grpc.health.v1.Health>", host: "<redacted>"
10.244.1.1 - - [20/Dec/2023:18:57:56 +0000] "GET /grpc.health.v1.Health HTTP/2.0" 009 15 "-" "curl/8.1.2" 56 0.002 [flyte-flyte-flyte-binary-grpc-8089] [] 10.244.1.83:8089 15 0.001 200 8a9b5ebf0300d82491b7d06e1b392f3f

I don’t see any related loglines when I check the logs for the flyte-binary pod. My

~/.flyte/config.yaml

looks like

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///<domain>
  insecure: false
  authType: Pkce
console:
  endpoint: <domain>

Questions: 1. Does anyone have any pointers here? 2. I’m surprised to not see any logs related to the error when I

kubectl -n flyte logs flyte-binary

. Curious if anyone has any pointers for debugging these sorts of issues. Where are the logs? Thanks, community! 🙏

can you check the annotations for your ingress resource? Something like this has proven to work in the past:

commonAnnotations:
    <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  host: <your-Flyte-URL>

Should

grpcAnnotations.host

have the protocol or just the domain? (i.e.,

<https://flyte.my.domain.com>

vs.

<http://flyte.my.domain.com|flyte.my.domain.com>

)

host

is not under

grpc.Annotations

. And it should be just the domain without https

Great, thanks! Trying now.

if it doesn't work, try adding the following flag to your

config.yaml

admin
  ...
  insecureSkipVerify:true

How are you issuing the certificate for the Ingress? maybe cert-manager or?

Yep, cert-manager.

yep, that looks alright

💥 Thanks!

There's a PR in place that updates the recommended annotations: https://github.com/flyteorg/flyte/pull/4604 Feel free to comment or keep improving the deployment guides! Recently, there have been efforts towards automating the entire process( see here) but there's a lot to be done 🙂

👀