curved-easter-24577
09/21/2023, 7:06 PMflytesnacks git:(master) pyflyte run --remote examples/basics/basics/hello_world.py hello_world_wf
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
details: failed to connect to all addresses; last error: INTERNAL: ipv4:10.1.64.90:443: Trying to connect an http1.x server
Debug string UNKNOWN:failed to connect to all addresses; last error: INTERNAL: ipv4:10.1.64.90:443: Trying to connect an http1.x server {grpc_status:14, created_time:"2023-09-21T20:46:30.828791+02:00"}
We can access the console UI with the our DNS but the GRPC we do not have access to. Our flyte config is:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///<our dns>
authType: Pkce
insecureSkipVerify: true # only required if using a self-signed cert. Caution: not to be used in production
insecure: true # only required when using insecure ingress. Secure ingress may cause an unavailable desc error to true option.
logger:
show-source: true
level: 6
Any idea what is happening? Thank you ❤️curved-easter-24577
09/21/2023, 7:06 PMingress:
albSSLRedirect: true
separateGrpcIngress: true
annotations:
# -- aws-load-balancer-controller v2.1 or higher is required - https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/
# For EKS if using [ALB](https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/ingress/annotations/), these annotations are set
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/tags: service_instance=production
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/security-groups: $CLUSTER_SG,$CLUSTER_CUSTOM_SG
alb.ingress.kubernetes.io/subnets: $CLUSTER_SUBNETS
alb.ingress.kubernetes.io/target-type: ip
# -- This is the certificate arn of the cert imported in AWS certificate manager.
alb.ingress.kubernetes.io/certificate-arn: "{{ .Values.userSettings.certificateArn }}"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
# -- Instruct ALB Controller to not create multiple load balancers (and hence maintain a single endpoint for both GRPC and Http)
alb.ingress.kubernetes.io/group.name: flyte
separateGrpcIngressAnnotations:
alb.ingress.kubernetes.io/backend-protocol-version: GRPCcurved-easter-24577
09/21/2023, 7:10 PMName: flyte-core-grpc
Labels: app.kubernetes.io/managed-by=Helm
Namespace: flyte
Address: <our aws load balancer>
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/flyteidl.service.SignalService flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.SignalService/* flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.AdminService flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.AdminService/* flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.DataProxyService flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.DataProxyService/* flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.AuthMetadataService flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.AuthMetadataService/* flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.IdentityService flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/flyteidl.service.IdentityService/* flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/grpc.health.v1.Health flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
/grpc.health.v1.Health/* flyteadmin:81 (10.1.52.244:8089,10.1.56.226:8089)
Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:
{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/backend-protocol-version: GRPC
alb.ingress.kubernetes.io/certificate-arn: <our cert arn>
alb.ingress.kubernetes.io/group.name: flyte
alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}]
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/security-groups: <our cluster sg>
alb.ingress.kubernetes.io/subnets: <our cluster subnets>
alb.ingress.kubernetes.io/tags: service_instance=production
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: flyte
meta.helm.sh/release-namespace: flyte
nginx.ingress.kubernetes.io/app-root: /console
nginx.ingress.kubernetes.io/backend-protocol: GRPC
Events: <none>curved-easter-24577
09/21/2023, 7:10 PMaverage-finland-92144
09/21/2023, 8:26 PMinsecure: false in your config filecurved-easter-24577
09/21/2023, 10:57 PM