I m having issues accessing the grpc endpoint behind an AWS Flyte #ask-the-community

I'm having issues accessing the grpc endpoint behi...

Rob Rati

05/30/2023, 3:19 PM

I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by following this: https://docs.flyte.org/en/latest/deployment/deployment/cloud_production.html And we have the http endpoint exposed via alb w/o issue. We use the aws ingress controller to create the ALBs, so I've modified the grpc ingress definition to add the following annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP alb.ingress.kubernetes.io/backend-protocol-version: GRPC alb.ingress.kubernetes.io/certificate-arn: <redacted> alb.ingress.kubernetes.io/healthcheck-path: /grpc.health.v1.Health alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: EKS-ALB-sg alb.ingress.kubernetes.io/target-type: ip And the spec with: ingressClassName: alb rules: - host: <http://flyte-web-east2-grpc.api-platform-dev.aws-nonprod.fmrcloud.com|<>hostname> But whenever I try to use grpcurl to access the endpoint to list/describe I always get: $ grpcurl <hostname>:443 list Failed to list services: server does not support the reflection API $ grpcurl <hostname>:443 describe /flyteidl.service.IdentityService Failed to resolve symbol "/flyteidl.service.IdentityService": server does not support the reflection API When I try to use grpc_cli ls <hostname>:443 I get an error: Trying to connect an http1.x server I can describe the grpc endpoints when I port-forward the grpc service, so the server is working as expected. Has anyone else tried to do this?

David Espejo (he/him)

05/30/2023, 3:57 PM

Hi @Rob Rati In this guide, there are annotations that work with the most recent

flyte-binary

chart version using ALB: https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/08-adjust-values-upgrade-Helm.md

jeev

05/30/2023, 3:58 PM

GRPC needs http2 that only works over SSL (443) on ALB afaik.

Rob Rati

05/30/2023, 3:58 PM

I have ssl to the ALB, but atm it's http/unsecured from the ALB to flyte

jeev

05/30/2023, 4:00 PM

i believe you also need: alb.ingress.kubernetes.io/backend-protocol-version: GRPC as an annotation on the GPRC-only ingress

Rob Rati

05/30/2023, 4:01 PM

I have that

Rob Rati

05/30/2023, 4:02 PM

I also think I have all the annotations in that "flyte the hard way" doc. Slight difference in that the scheme is internal instead of internet-facing, but otherwise I think I have (or have tried) all the options in that doc

Rob Rati

05/30/2023, 4:03 PM

Unless the group.name is important? That's for sharing an ingress afaik, and we only have 1 flyte instance we are working on atm

jeev

05/30/2023, 4:17 PM

can you try ONLY the following annotations:

Copy code

<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: '<ARN>'
<http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}]'
<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internal
<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb

for just the GRPC ingress? the HTTP endpoints should be in a different ingress

Rob Rati

05/30/2023, 4:18 PM

Sure, yas. For clarity, Http endpoints are in a different ingress and that ingress has been exposed via a different ALB w/o problem.

jeev

05/30/2023, 4:19 PM

my intuition is that the issue is with this annotation:

Copy code

<http://alb.ingress.kubernetes.io/backend-protocol|alb.ingress.kubernetes.io/backend-protocol>: HTTP

jeev

05/30/2023, 4:20 PM

i added:

<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'

in there for parity, but not sure its necessary since you are only listening on 443

Rob Rati

05/30/2023, 4:22 PM

I've tried with and w/o the backend-protocol. It's my understanding from reading that if you don't define it, then it defaults to HTTP. I just re-configured with just those annotations (and some labels we need to add)

Rob Rati

05/30/2023, 4:23 PM

Same errors. 😞

jeev

05/30/2023, 4:37 PM

ah yea you're right

Rob Rati

05/30/2023, 4:38 PM

I shouldn't have to do https from the alb -> flyte, right? I'll need to if we move forward, but for now just trying to test

jeev

05/30/2023, 4:39 PM

i dont think so

jeev

05/30/2023, 4:39 PM

can you

kubectl get ingress

jeev

05/30/2023, 4:39 PM

are the hostnames different for the 2 ingresses?

Rob Rati

05/30/2023, 4:41 PM

Yes

jeev

05/30/2023, 4:42 PM

looking at this: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#ingressgroup implying there are multiple ALBs right now.

Rob Rati

05/30/2023, 4:43 PM

An ingress group allows you to share 1 ingress with multiple backend applications that are the same or very, very similar. So, if you wanted to have say 5 flyte instances running, you can share an alb rather than having 5 different albs

jeev

05/30/2023, 4:46 PM

but if you have multiple ALBs, which ALB do you route your host to?

jeev

05/30/2023, 4:47 PM

when you do

kubectl get ingress

are the ALB addresses different?

Rob Rati

05/30/2023, 4:48 PM

Right now we have 2 albs. One for the http traffic, and another for grpc. This probably isn't workable pattern for flyte, but atm we haven't gotten there yet. Just trying to get grpc access working through the ingress via grpcurl or grpc_cli. Once we have that it'll be easy to combine the two ingresses into a single one. Unless you think having them separate is causing problems?

jeev

05/30/2023, 4:49 PM

unfortunately, we need to keep them separate in this case because of the GRPC-specific annotations

Rob Rati

05/30/2023, 4:49 PM

Yes, right now the two albs (via kubectl get ingress) have different alb addresses

jeev

05/30/2023, 4:49 PM

but i think the group.name will allow them to be merged into a single ALB

Rob Rati

05/30/2023, 4:49 PM

Correct

Rob Rati

05/30/2023, 4:49 PM

I just haven't done that step yet. 😄

jeev

05/30/2023, 4:50 PM

in the code snippet above:

<hostname>:443

were you using the ALB address or your internal hostname?

Rob Rati

05/30/2023, 4:50 PM

I figured I could verify grpc functionality first to try to reduce the potential causes of problems

Rob Rati

05/30/2023, 4:51 PM

ALB address. I'm accessing from my laptop, not a pod on the eks cluster.

jeev

05/30/2023, 4:51 PM

i see

Rob Rati

05/30/2023, 4:51 PM

I'm using the same kind of url as we use for the http, which successfully gets us to flyte's console

Rob Rati

05/30/2023, 4:53 PM

Although, just noticed that ports listed in

kubectl get ingress

for the grpc is port 80. I would think it should be port 443?

Rob Rati

05/30/2023, 4:54 PM

% kubectl -n morgan-test get ingress NAME CLASS HOSTS ADDRESS PORTS AGE grpc alb <url> <http://internal-k8s-morgante-morgante-73da0a0e06-1041092043.us-east-2.elb.amazonaws.com|<>internal_url> 80 5d23h http alb <http://flyte-web-east2.api-platform-dev.aws-nonprod.fmrcloud.com|<>url> <http://internal-k8s-morgante-morgante-7641a7c2a3-1671587638.us-east-2.elb.amazonaws.com|<>internal_url> 80 10d

jeev

05/30/2023, 4:59 PM

im seeing the same in my test cluster as well

Rob Rati

05/30/2023, 5:09 PM

Same errors? Are you running https from alb->flyte or http?

jeev

05/30/2023, 5:11 PM

alb -> flyte

Rob Rati

05/30/2023, 5:12 PM

I mean what protocol between alb->flyte. http or https?

jeev

05/30/2023, 5:14 PM

http

jeev

05/30/2023, 5:14 PM

try this:

Copy code

flytectl get project --admin.endpoint=<hostname>:443

Rob Rati

05/30/2023, 5:17 PM

That won't need to use http right? That's a grpc only call?

jeev

05/30/2023, 5:17 PM

yea thats a grpc call

jeev

05/30/2023, 5:17 PM

you can confirm with a port-forward of just the grpc port and using

localhost:8089

Rob Rati

05/30/2023, 5:20 PM

Well, flytectl seems to have worked? $ flytectl get project --admin.endpoint=<hostname>:443 INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. ------------- ------------- ------------------------- | ID | NAME | DESCRIPTION | ------------- ------------- ------------------------- | flytesnacks | flytesnacks | flytesnacks description | ------------- ------------- ------------------------- 1 rows

jeev

05/30/2023, 5:20 PM

cool

jeev

05/30/2023, 5:20 PM

so your ingress is good

Rob Rati

05/30/2023, 5:21 PM

Huh. I guess I didn't think I could use grpcurl on a portforward but not through the ingress

jeev

05/30/2023, 5:22 PM

can you paste your grpcurl command, and output?

jeev

05/30/2023, 5:22 PM

i think it might be because we're not exposing all the grpc methods in the ingress. but not 100% sure.

Rob Rati

05/30/2023, 5:22 PM

grpcurl <hostname>:443 describe /flyteidl.service.IdentityService

Rob Rati

05/30/2023, 5:22 PM

That endpoint is one that is listed in the ingress definition

jeev

05/30/2023, 5:24 PM

Copy code

server does not support the reflection API

suggests there are reflection specific methods, perhaps?

jeev

05/30/2023, 5:25 PM

see: https://github.com/solo-io/gloo/issues/3244#issue-647618923

jeev

05/30/2023, 5:26 PM

specifcally:

Copy code

grpc.reflection.v1alpha.ServerReflection

Rob Rati

05/30/2023, 5:26 PM

I have been looking around on this, and it seems the server needs to do somethings to support reflection. I thought all that was done though since I can do the grpcurl call through the service when it is port-forwarded. So, since I can do an explain over port-forward, I thought the server supported reflection and I should be able to do the same through the ingress

Rob Rati

05/30/2023, 5:27 PM

Not a grpc expert at all though. 😄

jeev

05/30/2023, 5:27 PM

right but the ingress doesnt include that route

jeev

05/30/2023, 5:27 PM

ill try and get that working

Rob Rati

05/30/2023, 5:27 PM

Rob Rati

05/30/2023, 5:28 PM

Reflection is a different route?

jeev

05/30/2023, 5:29 PM

Copy code

> docker run fullstorydev/grpcurl -v -plaintext host.docker.internal:8089 list
flyteidl.service.AdminService
flyteidl.service.AuthMetadataService
flyteidl.service.DataProxyService
flyteidl.service.IdentityService
flyteidl.service.SignalService
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection

Rob Rati

05/30/2023, 5:30 PM

Oh, I see. I was seeing that error but didn't realize that was a different route

Rob Rati

05/30/2023, 5:30 PM

The information was all in front of me from the start

jeev

05/30/2023, 5:30 PM

we can just expose it in the ingress

Rob Rati

05/30/2023, 5:35 PM

TY for the help and your patience. :)

jeev

05/30/2023, 5:36 PM

hopefully someone will find this useful 🙂

Rob Rati

05/30/2023, 5:38 PM

One last question while we're on the subject. Are there docs for configuring alb<->flyte access via https?

jeev

05/30/2023, 5:59 PM

sorry idk about docs, but you should be just able to turn on ssl: https://github.com/flyteorg/flyteadmin/blob/15d5c7bb568b570890b1c696f0d01466ea6bef2d/pkg/config/config.go#L23 https://github.com/flyteorg/flyteadmin/blob/15d5c7bb568b570890b1c696f0d01466ea6bef2d/pkg/config/config.go#LL69C37-L69C37

890 Views

Open in Slack

Previous Next