@Fabio Grätz Hi Fabio, how’s it going? I have few questions regarding your workaround… 1. In your configuration, did you enable the

separateGrpcIngress

? I understand from the documentation that it is Required for certain ingress controllers like nginx. 2. After deploying nginx (which uses our own tls certificate, not self-signed with cert-manager), I’m getting 502 bad gateway errors accessing the console (flyte.my.domain)… In the

flyteadmin

logs I see this authentication error:

Failed to refresh tokens. Restarting login flow. Error: [TOKEN_REFRESH_FAILURE] Error refreshing token, caused by: oauth2: cannot fetch token: 400 Bad Request

(it worth mentioning I don’t see any errors in Okta, which we configured there the authorization server) 3. Regarding accessing within the CLI, while trying to use

flytectl

I’m still getting the same authentication error:

PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type

- Did you experience this thing? BTW, I don’t understand from your answer if you’re using

flytectl/pyflyte

? Thanks in advanced! And sorry for all the questions, I appreciate your help!

I unfortunately don’t find my nginx manifest from back then anymore (new computer and branch in gh was deleted).

separateGrpcIngress

rings a bell, I am pretty sure I had to enable this.

<http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'

is needed for flyteadmin (I don’t think for datacatalog since it is not exposed to the user?) Are you sure it’s

<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP

and not

<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"

? (source)

Thank you for your answer!

Awesome, nice to hear 🙂

Thank you so much @Fabio Grätz! Also @Ariel Kaspit if you're short on time and could share your ingress config, I can work on a PR and add you as co-author to give you the credit 🙂

Yes! Thank you @David Espejo (he/him) Actually I didn’t have much to configure… For the

flyte

chart configuration it’s exactly as mentioned in

values-gcp.yaml

. I enabled the separated ingress option:

common:
            ingress:
              host: "flyte.my.domain"
              annotations:
                <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
                <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
              separateGrpcIngress: true
              separateGrpcIngressAnnotations:
                <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"

For the

nginx-ingress-controller

helm chart configuration, please note I used a certificate from Cloudflare (not cert-manager) and I enabled gRPC networking in Cloudflare account (that was the reason for getting

403 forbidden

errors while accessing `flytectl`/`pyflyte`)

controller:
          service:
            annotations:
              <http://external-dns.alpha.kubernetes.io/hostname|external-dns.alpha.kubernetes.io/hostname>: flyte.my.domain
          config:
            proxy-buffer-size: "16k"
          extraArgs:
            default-ssl-certificate: "namespace/secret-tls-name"

Do you do the PR @David Espejo (he/him)?

sure!