https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Art Gillespie

12/20/2023, 7:04 PM
Trying to move our deployment beyond port forwarding. I followed the instructions here and got everything working correctly for the console/UI with TLS, but I’m having difficulty with
flytectl
. When I run
flytectl -n flytesnacks -p development get tasks
I get
Copy code
Error: Connection Info: [Endpoint: dns:///<redacted>, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type
{"json":{},"level":"error","msg":"Connection Info: [Endpoint: dns:///<redacted>, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type","ts":"2023-12-20T10:48:53-08:00"}
When I try curl on the
grpc.health.v1.Health
endpoint
Copy code
*   Trying <redacted>...
* Connected to <redacted> (<redacted>) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=<redacted>
*  start date: Dec 20 14:30:05 2023 GMT
*  expire date: Mar 19 14:30:04 2024 GMT
*  subjectAltName: host "<redacted>" matched cert's "<redacted>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: <redacted>]
* h2 [:path: /grpc.health.v1.Health]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x11d814200)
> GET /grpc.health.v1.Health HTTP/2
> Host: <redacted>
> User-Agent: curl/8.1.2
> Accept: */*
>
* Unsupported response code in HTTP response
* Connection #0 to host <redacted> left intact
When I check the ingress-nginx controller’s logs I see
Copy code
2023/12/20 18:57:56 [error] 241#241: *3348260 recv() failed (104: Connection reset by peer) while reading upstream, client: 10.244.1.1, server: <redacted>, request: "GET /grpc.health.v1.Health HTTP/2.0", upstream: "<http://10.244.1.83:8089/grpc.health.v1.Health>", host: "<redacted>"
10.244.1.1 - - [20/Dec/2023:18:57:56 +0000] "GET /grpc.health.v1.Health HTTP/2.0" 009 15 "-" "curl/8.1.2" 56 0.002 [flyte-flyte-flyte-binary-grpc-8089] [] 10.244.1.83:8089 15 0.001 200 8a9b5ebf0300d82491b7d06e1b392f3f
I don’t see any related loglines when I check the logs for the flyte-binary pod. My
~/.flyte/config.yaml
looks like
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///<domain>
  insecure: false
  authType: Pkce
console:
  endpoint: <domain>
Questions: 1. Does anyone have any pointers here? 2. I’m surprised to not see any logs related to the error when I
kubectl -n flyte logs flyte-binary
. Curious if anyone has any pointers for debugging these sorts of issues. Where are the logs? Thanks, community! 🙏
Oh, and we’re running Google OIDC for authentication. From our flyte-binary chart values:
Copy code
auth:
    enabled: true
    oidc:
      baseUrl: <https://accounts.google.com>
      clientId:  {{ .Values.secrets.flyte.auth.oidc.clientId }}
      clientSecret:  {{ .Values.secrets.flyte.auth.oidc.clientSecret }}
    internal:
      clientSecret: {{ .Values.secrets.flyte.auth.internal.clientSecret }}
      clientSecretHash:  {{ .Values.secrets.flyte.auth.internal.clientSecretHash }}
    authorizedUris:
    - https://<domain>
d

David Espejo (he/him)

12/20/2023, 8:05 PM
can you check the annotations for your ingress resource? Something like this has proven to work in the past:
Copy code
commonAnnotations:
    <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  host: <your-Flyte-URL>
a

Art Gillespie

12/20/2023, 8:09 PM
Should
grpcAnnotations.host
have the protocol or just the domain? (i.e.,
<https://flyte.my.domain.com>
vs.
<http://flyte.my.domain.com|flyte.my.domain.com>
)
d

David Espejo (he/him)

12/20/2023, 8:10 PM
host
is not under
grpc.Annotations
. And it should be just the domain without https
a

Art Gillespie

12/20/2023, 8:11 PM
Great, thanks! Trying now.
d

David Espejo (he/him)

12/20/2023, 8:12 PM
if it doesn't work, try adding the following flag to your
config.yaml
Copy code
admin
  ...
  insecureSkipVerify:true
How are you issuing the certificate for the Ingress? maybe cert-manager or?
a

Art Gillespie

12/20/2023, 8:13 PM
Yep, cert-manager.
chart values look like
Copy code
ingress:
  create: true
  host: flyte.<my-domain>
  tls:
  - hosts: 
      - flyte.<my-domain>
    secretName: "flyte-tls"
  commonAnnotations:
    <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-prod"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  ingressClassName: nginx
d

David Espejo (he/him)

12/20/2023, 8:14 PM
yep, that looks alright
a

Art Gillespie

12/20/2023, 8:14 PM
💥 Thanks!
Works. Amazing. Thanks so much for your help, David! 🙏
Did I miss this in the docs somewhere, or is writing this up something I could contribute back?
d

David Espejo (he/him)

12/20/2023, 8:18 PM
There's a PR in place that updates the recommended annotations: https://github.com/flyteorg/flyte/pull/4604 Feel free to comment or keep improving the deployment guides! Recently, there have been efforts towards automating the entire process( see here) but there's a lot to be done 🙂
a

Art Gillespie

12/20/2023, 8:20 PM
👀
2 Views