Ethan Brown
11/29/2023, 12:18 AMDavid Espejo (he/him)
11/29/2023, 3:41 PMsync-resources
Pod. If, after configuring everything, that Pod is Running
, that's a good signEthan Brown
11/29/2023, 3:43 PMflytepropeller
inside the data plane. Jobs look to be sent over to the data plane properly, but it looks to me like traffic isn't properly flowing out of that cluster over GRPC back to the control plane.
Double-checking my ingress definition now (and my ingress logs)Ethan Brown
11/29/2023, 3:45 PM{"json":{"exec_id":"f194818b7dbea4957858","ns":"flytesnacks-development","res_ver":"10430815","routine":"worker-1","wf":"flytesnacks:development:.flytegen.basic-task.slope"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"]]","ts":"2023-11-29T15:41:06Z"}
{"json":{"exec_id":"f194818b7dbea4957858","ns":"flytesnacks-development","res_ver":"10430815","routine":"worker-1","wf":"flytesnacks:development:.flytegen.basic-task.slope"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]","ts":"2023-11-29T15:41:06Z"}
David Espejo (he/him)
11/29/2023, 3:46 PMEthan Brown
11/29/2023, 3:46 PMDavid Espejo (he/him)
11/29/2023, 3:47 PMEthan Brown
11/29/2023, 3:47 PMDavid Espejo (he/him)
11/29/2023, 3:50 PMcommon:
ingress:
host: "{{ .Values.userSettings.hostName }}"
tls:
enabled: true
secretName: flyte-secret-tls
annotations:
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
<http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
<http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
<http://acme.cert-manager.io/http01-edit-in-place|acme.cert-manager.io/http01-edit-in-place>: "true"
# --- separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx.
separateGrpcIngress: true
# --- Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled.
separateGrpcIngressAnnotations:
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
You can remove the cert-manager related contentEthan Brown
11/29/2023, 3:51 PMEthan Brown
11/29/2023, 3:53 PMconfigmap:
admin:
admin:
endpoint: publichost.domain.com:443
insecure: false
catalog:
catalog-cache:
endpoint: publichost.domain.com:443
insecure: false
Ethan Brown
11/29/2023, 4:07 PMgrpcurl
from outside the ingress to validate things?David Espejo (he/him)
11/29/2023, 4:15 PMEthan Brown
11/29/2023, 8:40 PMgrpcurl -plaintext flyteadmin.flyte:81 list
Ethan Brown
11/29/2023, 8:40 PM