Hey Team, I have everything I need working properly, except external access via ingress. I added t...

fierce-match-73373

12/15/2023, 4:47 PM

Hey Team, I have everything I need working properly, except external access via ingress. I added this ingress section to my local-values.yaml file for a flyte-binary helm deployment on k3s:

ingress:

create: true

commonAnnotations:

<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: traefik

httpAnnotations:

<http://traefik.ingress.kubernetes.io/app-root|traefik.ingress.kubernetes.io/app-root>: /console

grpcAnnotations:

<http://ingress.kubernetes.io/protocol|ingress.kubernetes.io/protocol>: h2c

<http://traefik.ingress.kubernetes.io/backend-protocol|traefik.ingress.kubernetes.io/backend-protocol>: h2c

<http://ingress.kubernetes.io/backend-protocol|ingress.kubernetes.io/backend-protocol>: h2c

host: <my fqdn>

Once I reload with "helm update", the web interface works perfectly on the host fqdn, however the gRPC system doesn't seem to function. I can't submit jobs with pyflyte, or do other command line related actions. I get the following error when using pyflyte register:

E1215 09:24:17.346769202 250039 <http://hpack_parser.cc:999]|hpack_parser.cc:999]>         Error parsing 'content-type' metadata: invalid value

Failed with Exception Code: SYSTEM:Unknown

RPC Failed, with Status: StatusCode.UNKNOWN
details: Stream removed

Debug string UNKNOWN:Error received from peer {grpc_message:"Stream removed", grpc_status:2, created_time:"2023-12-15T09:24:17.372600113-06:00"}

Any ideas? I've attempted debugging by viewing the logs, but I don't see errors in any of the traefik logs, any way to debug more verbosely? Removing the ingress section returns everything to a fully functional state (after re-forwarding ports 8088/8089 as specified the the guide).

better-lamp-53311

02/21/2024, 11:54 AM

Hi garretcook , did you solve this issue? We are at the exact same spot right now.

15 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.