https://flyte.org logo
Join Slack
Powered by
<@U04664Z7H37> Hi Fabio, how’s it going? I have fe...
# flyte-on-gcp
a
@Fabio Grätz Hi Fabio, how’s it going? I have few questions regarding your workaround… 1. In your configuration, did you enable the 
separateGrpcIngress
? I understand from the documentation that it is Required for certain ingress controllers like nginx. 2. After deploying nginx (which uses our own tls certificate, not self-signed with cert-manager), I’m getting 502 bad gateway errors accessing the console (flyte.my.domain)… In the 
flyteadmin
logs I see this authentication error: 
Failed to refresh tokens. Restarting login flow. Error: [TOKEN_REFRESH_FAILURE] Error refreshing token, caused by: oauth2: cannot fetch token: 400 Bad Request
(it worth mentioning I don’t see any errors in Okta, which we configured there the authorization server) 3. Regarding accessing within the CLI, while trying to use 
flytectl
I’m still getting the same authentication error: 
PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
- Did you experience this thing? BTW, I don’t understand from your answer if you’re using 
flytectl/pyflyte
? Thanks in advanced! And sorry for all the questions, I appreciate your help!
I also tried to use 
pyflyte
and the error is:
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.PERMISSION_DENIED
        details = "Received http2 header with status: 403"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received http2 header with status: 403", grpc_status:7, created_time:"2023-05-31T19:41:03.999452+03:00"}"
It seems to still have issues with the gRPC service… Did I miss something?
I followed the values-gcp.yaml in Flyte repository. Adding 
flyteadmin
and 
datacatalog
this following annotation; 
<http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
Also, added these following annotations to the ingress: 
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
f
I unfortunately don’t find my nginx manifest from back then anymore (new computer and branch in gh was deleted). 
separateGrpcIngress
rings a bell, I am pretty sure I had to enable this. 
<http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
is needed for flyteadmin (I don’t think for datacatalog since it is not exposed to the user?) Are you sure it’s 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP
and not 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
? (source)
I think what makes it harder is that in the current situation it’s not easy to tell whether the problem is caused by the gRPC service or by auth. Do you by any chance have access to some sandbox environment where you can try without auth for a short time to rule out one of the two?
Also, I only tried with cert manager, never with self-signed certificate, not sure which kind of influence this might have.
Did you experience this thing? BTW, I don’t understand from your answer if you’re using 
flytectl/pyflyte
?
We use both sporadically but have our own script based on flyte remote. However, in our current setup we cannot register … workflows with CLIs through the ingress because it uses IAP and we have not invested the time yet to try to make this work. Since all of our engineers work on VMs almost exclusively, we in the end gave flyteadmin a 2nd internal load balancer type service that is reachable from within the VPC.
a
Thank you for your answer!
Ok, so I solved the issue. Thank you for your help again… I’m using nginx ingress controller and enabled the 2 ingress separation for flyte and flyte-admin with the right annotations for each one of them. The console works and the errors I got when accessing from 
flytectl
and 
pyflyte
(
unexpected HTTP status code received from server: 403 (Forbidden)
) came from Cloudflare. It was solved after enabling gRPC.
f
Awesome, nice to hear 🙂
Would you be willing to open a PR in the docs with the changes to the nginx ingress you had to do? If yes you can tag me as reviewer.
Would be nice to persist this knowledge
d
Thank you so much @Fabio Grätz! Also @Ariel Kaspit if you're short on time and could share your ingress config, I can work on a PR and add you as co-author to give you the credit 🙂
a
Yes! Thank you @David Espejo (he/him) Actually I didn’t have much to configure… For the 
flyte
chart configuration it’s exactly as mentioned in 
values-gcp.yaml
. I enabled the separated ingress option:
Copy code
common:
            ingress:
              host: "flyte.my.domain"
              annotations:
                <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
                <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
              separateGrpcIngress: true
              separateGrpcIngressAnnotations:
                <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
For the 
nginx-ingress-controller
helm chart configuration, please note I used a certificate from Cloudflare (not cert-manager) and I enabled gRPC networking in Cloudflare account (that was the reason for getting 
403 forbidden
errors while accessing `flytectl`/`pyflyte`)
Copy code
controller:
          service:
            annotations:
              <http://external-dns.alpha.kubernetes.io/hostname|external-dns.alpha.kubernetes.io/hostname>: flyte.my.domain
          config:
            proxy-buffer-size: "16k"
          extraArgs:
            default-ssl-certificate: "namespace/secret-tls-name"
f
Do you do the PR @David Espejo (he/him)?
Happy to be tagged as reviewer
d
sure!
cc @Haytham Amin
184 Views
Open in Slack
PreviousNext
Powered by