Fabio Grätz
06/28/2023, 4:01 PM❯ pyflyte run --remote wf.py wf
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
details: failed to connect to all addresses; last error: INTERNAL: ipv4:<http://104.xxx:443|104.xxx:443>: Trying to connect an http1.x server
Debug string UNKNOWN:failed to connect to all addresses; last error: INTERNAL: ipv4:104.198.187.232:443: Trying to connect an http1.x server {created_time:"2023-06-28T18:01:30.030524+02:00", grpc_status:14}
The nginx controller shows this log line for this request:
<http://10.xxx|10.xxx> - - [28/Jun/2023:16:01:30 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.132 [] [] - - - - fb2516a8919e168c4a8ef36f31f3945a
admin:
endpoint: dns:///<my-domain>.com
insecure: true
insecureSkipVerify: true
logger:
show-source: true
level: 0
storage:
type: stow
stow:
kind: google
config:
json: ""
project_id: <my-gcp-project-id>
scopes: <https://www.googleapis.com/auth/devstorage.read_write>
common:
ingress:
host: "{{ .Values.userSettings.hostName }}"
tls:
enabled: true
annotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
<http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-prod"
separateGrpcIngress: true
separateGrpcIngressAnnotations:
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
David Espejo (he/him)
06/28/2023, 4:42 PMinsecure
and insecureSkipVerify
set to true
have an effect.
What if you set
insecure: false
?Tommy Nam
06/29/2023, 3:33 AMingress:
create: true
host: <http://customhostname.com|customhostname.com>
httpAnnotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
<http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}, {"HTTP":80}]'
<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>:
<http://alb.ingress.kubernetes.io/ssl-policy|alb.ingress.kubernetes.io/ssl-policy>: ELBSecurityPolicy-TLS-1-1-2017-01 #Optional (Picks default if not used)
<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
<http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
<http://alb.ingress.kubernetes.io/group.order|alb.ingress.kubernetes.io/group.order>: '10'
grpcAnnotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
<http://alb.ingress.kubernetes.io/backend-protocol|alb.ingress.kubernetes.io/backend-protocol>: HTTP
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
<http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}]'
<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>:
<http://alb.ingress.kubernetes.io/ssl-policy|alb.ingress.kubernetes.io/ssl-policy>: ELBSecurityPolicy-TLS-1-1-2017-01
<http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
<http://alb.ingress.kubernetes.io/group.order|alb.ingress.kubernetes.io/group.order>: '20'
Just an example of our ingress setup on the flyte-binary helm chart. Hopefully this will shed some light on your issueFabio Grätz
06/29/2023, 2:33 PMinsecure: false
, thanks @David Espejo (he/him)! I could have sworn this flag controlled whether auth is used or not 🤦♂️David Espejo (he/him)
06/29/2023, 2:34 PMFabio Grätz
06/29/2023, 2:34 PM