Has anyone seen this issue when connecting to flyte GRPC over TLS (i.e insecure: false). I get this ...

curved-whale-1505

04/30/2025, 7:19 AM

Has anyone seen this issue when connecting to flyte GRPC over TLS (i.e insecure: false). I get this error when just running “pyflyte info”. I verified I am able to use grpcurl to hit the server successfully but when using pyflyte I get this error:

Cannot check peer: missing selected ALPN property.

curved-whale-1505

04/30/2025, 7:32 AM

Fixed it! I had to set the ALPN Policy on my network load balancer. I chose to set it to

HTTP2Only

which did the trick. Sources: • https://aws.amazon.com/about-aws/whats-new/2020/05/network-load-balancer-now-supports-tls-alpn-policies/ • https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/service/annotations/#alpn-policy

curved-whale-1505

04/30/2025, 8:08 AM

Follow up question: Now that I have

pyflyte info

working properly over TLS, I get this error at the end of every command:

Copy code

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1746000428.517432   35510 <http://init.cc:229]|init.cc:229]> grpc_wait_for_shutdown_with_timeout() timed out.

Any ideas what causes this error? Wondering if there's some other setting I need to change on the load balancer.

curved-whale-1505

04/30/2025, 8:13 AM

I guess we need to pull in this latest commit of the grpc package? https://github.com/grpc/grpc/commit/6b0f95b350f623c29ad24a4695a9aa2e8d0b1957

careful-australia-19356

05/02/2025, 6:56 PM

I guess you're running with a higher

logger

level? Otherwise I cannot repro that message

careful-australia-19356

05/02/2025, 6:56 PM

does workflow executions work for you?

curved-whale-1505

05/02/2025, 10:59 PM

yeah, workflow executions do indeed work for me

curved-whale-1505

05/02/2025, 11:00 PM

i am not changing the log level as far as i know (not sure how)

curved-whale-1505

05/02/2025, 11:01 PM

i only get the log message when using flyte grpc over TLS. when running insecure via port forwarding I don’t see this message

29 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.