https://flyte.org logo
Join Slack
Powered by
Hey everyone, I need some help with debugging why...
# ask-the-community
f
Hey everyone, I need some help with debugging why accessing flyteadmin through my nginx ingress doesn’t work. Details in 🧵 (Background: We run Flyte without auth behind a GKE Ingress with a GCP identity aware proxy using istio. Flyteadmin is available within the VPC from where we register. I want to explore whether I can make Flyte work in authenticated mode with GKE ingress and with GCP identity aware proxy. This would be my dream setup for GCP. For that I first want to have a working flyte deployment with authentication and nginx ingress.)
Pyflyte gives me this error:
Copy code
❯ pyflyte run --remote wf.py wf
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
        details: failed to connect to all addresses; last error: INTERNAL: ipv4:<http://104.xxx:443|104.xxx:443>: Trying to connect an http1.x server
        Debug string UNKNOWN:failed to connect to all addresses; last error: INTERNAL: ipv4:104.198.187.232:443: Trying to connect an http1.x server {created_time:"2023-06-28T18:01:30.030524+02:00", grpc_status:14}
The nginx controller shows this log line for this request:
Copy code
<http://10.xxx|10.xxx> - - [28/Jun/2023:16:01:30 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.132 [] [] - - - - fb2516a8919e168c4a8ef36f31f3945a
This is my flyte config:
Copy code
admin:
  endpoint: dns:///<my-domain>.com
  insecure: true
  insecureSkipVerify: true
logger:
  show-source: true
  level: 0
storage:
  type: stow
  stow:
    kind: google
    config:
      json: ""
      project_id: <my-gcp-project-id>
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
This is the ingress section of my helm values:
Copy code
common:
  ingress:
    host: "{{ .Values.userSettings.hostName }}"
    tls:
      enabled: true
    annotations:
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
      <http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-prod"
    separateGrpcIngress: true
    separateGrpcIngressAnnotations:
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
I can reach the Flyte console on my domain and TLS is working correctly for it. So the certificate should be fine.
@David Espejo (he/him) do you have any idea? (Asking because you recently published your very nice “flyte the hard way” repo where you did exactly this for the alb ingress)
d
So if you can reach the console but pyflyte complains, it could be something in your config file I'm not sure both 
insecure
and 
insecureSkipVerify
set to 
true
have an effect. What if you set 
insecure: false
?
t
@Fabio Grätz We ran into slightly similar issues in the past and for us it was resolved by setting up our LoadBalancer to handle both GRPC and HTTP/S requests at the same host name/DNS address. In particular, we set the weight for the GRPC ingress manifest to be higher than the HTTP ingress. I'm by no means an expert but essentially I think the main problem is that your loadbalancer/ingress may not be properly setup to handle GRPC traffic
Copy code
ingress:
  create: true
  host: <http://customhostname.com|customhostname.com>
  httpAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}, {"HTTP":80}]'
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: 
    <http://alb.ingress.kubernetes.io/ssl-policy|alb.ingress.kubernetes.io/ssl-policy>: ELBSecurityPolicy-TLS-1-1-2017-01 #Optional (Picks default if not used)
    <http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
    <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
    <http://alb.ingress.kubernetes.io/group.order|alb.ingress.kubernetes.io/group.order>: '10'
  grpcAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/backend-protocol|alb.ingress.kubernetes.io/backend-protocol>: HTTP
    <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
    <http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}]'
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: 
    <http://alb.ingress.kubernetes.io/ssl-policy|alb.ingress.kubernetes.io/ssl-policy>: ELBSecurityPolicy-TLS-1-1-2017-01
    <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
    <http://alb.ingress.kubernetes.io/group.order|alb.ingress.kubernetes.io/group.order>: '20'
Just an example of our ingress setup on the flyte-binary helm chart. Hopefully this will shed some light on your issue
f
It really was 
insecure: false
, thanks @David Espejo (he/him)! I could have sworn this flag controlled whether auth is used or not 🤦‍♂️
Thanks @Tommy Nam too!
d
sorry for the confusions @Fabio Grätz. We'll have better docs
f
This one was on me I guess ^^
154 Views
Open in Slack
PreviousNext
Powered by