Hi, did anybody deployed flyte-core Helm chart on k8s using Kong as ingress? It seems that I have hit exactly this issue: https://kennethjenkins.net/posts/go-nginx-grpc/, it manifests when I'm trying top register a workflow using

flyteclt

. I wonder if there is a solution applicable on the ingress configuration side.

so are you getting that

too large http2 frame:

error when you try to register a workflow?

Yeah, I am getting exactly the error from that blog post:

| NAME                                                         | STATUS | ADDITIONAL INFO                                    |
 -------------------------------------------------------------- -------- ---------------------------------------------------- 
| /tmp/register76440984/0_workflows.hello_world.say_hello_1.pb | Failed | Error registering file due to rpc error: code =    |
|                                                              |        | Unavailable desc = connection error: desc = "error |
|                                                              |        | reading server preface: http2: frame too large"    |
 -------------------------------------------------------------- -------- ----------------------------------------------------

And I see matching errors in the ingress logs:

2024/08/08 11:24:57 [info] 1322#0: *3436854 client sent invalid request while reading client request line, client: 10.224.0.4, server: kong, request: "PRI * HTTP/2.0"

I got some hints in another thread, so at least I know this should be working, so I will try to see what am I doing wrong :).

yeah, depending on the controller, there are different annotations to be set. When you find the working config, it'd be great it you could share it, this should go into the docs

I was able to get it working, due to @victorious-jelly-28729’s help really. Went on vacation right afterwards, so now I can share. In my case there were two problems: I did not have TLS configured in Flyte (it was terminated on the ingress), and while Kong enables HTTP2 by default, it only enables it on the TLS-protected port. The second problem was that Kong apparently does not allow to expose both https(s) and grpc(s) ports on a single service, the way

flyteadmin

service it. This is because each service needs to be annotated with a protocol, and only one choice is supported. So I had to create a separate service - something that satya advised me, but I initially did not know why. For the time being I had to handle this with a workaround and manage the service "separately" becaue the helm chart does not support such a scenario.

@some-solstice-93243 thanks for sharing.

flyte-binary

supports two separate services, and AFAICR it was enabled to better support Kong

Oh, that's interesting. We were considering opening a PR to add this to

flyte-core

. Is there a reason it was only added to

flyte-binary

?

Not sure why, like other things, this was not also implemented on flyte-binary. I see that flyte-core supports separate Ingress resources but not separate service