I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by followi...

damp-potato-83989

05/30/2023, 3:19 PM

I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by following this: https://docs.flyte.org/en/latest/deployment/deployment/cloud_production.html And we have the http endpoint exposed via alb w/o issue. We use the aws ingress controller to create the ALBs, so I've modified the grpc ingress definition to add the following annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP alb.ingress.kubernetes.io/backend-protocol-version: GRPC alb.ingress.kubernetes.io/certificate-arn: <redacted> alb.ingress.kubernetes.io/healthcheck-path: /grpc.health.v1.Health alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: EKS-ALB-sg alb.ingress.kubernetes.io/target-type: ip And the spec with: ingressClassName: alb rules: - host: <http://flyte-web-east2-grpc.api-platform-dev.aws-nonprod.fmrcloud.com|<>hostname> But whenever I try to use grpcurl to access the endpoint to list/describe I always get: $ grpcurl <hostname>:443 list Failed to list services: server does not support the reflection API $ grpcurl <hostname>:443 describe /flyteidl.service.IdentityService Failed to resolve symbol "/flyteidl.service.IdentityService": server does not support the reflection API When I try to use grpc_cli ls <hostname>:443 I get an error: Trying to connect an http1.x server I can describe the grpc endpoints when I port-forward the grpc service, so the server is working as expected. Has anyone else tried to do this?

average-finland-92144

05/30/2023, 3:57 PM

Hi @damp-potato-83989 In this guide, there are annotations that work with the most recent

flyte-binary

chart version using ALB: https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/08-adjust-values-upgrade-Helm.md

freezing-boots-56761

05/30/2023, 3:58 PM

GRPC needs http2 that only works over SSL (443) on ALB afaik.

damp-potato-83989

05/30/2023, 3:58 PM

I have ssl to the ALB, but atm it's http/unsecured from the ALB to flyte

freezing-boots-56761

05/30/2023, 4:00 PM

i believe you also need: alb.ingress.kubernetes.io/backend-protocol-version: GRPC as an annotation on the GPRC-only ingress

damp-potato-83989

05/30/2023, 4:01 PM

I have that

damp-potato-83989

05/30/2023, 4:02 PM

I also think I have all the annotations in that "flyte the hard way" doc. Slight difference in that the scheme is internal instead of internet-facing, but otherwise I think I have (or have tried) all the options in that doc

damp-potato-83989

05/30/2023, 4:03 PM

Unless the group.name is important? That's for sharing an ingress afaik, and we only have 1 flyte instance we are working on atm

freezing-boots-56761

05/30/2023, 4:17 PM

can you try ONLY the following annotations:

Copy code

<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: '<ARN>'
<http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}]'
<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internal
<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb

for just the GRPC ingress? the HTTP endpoints should be in a different ingress

damp-potato-83989

05/30/2023, 4:18 PM

Sure, yas. For clarity, Http endpoints are in a different ingress and that ingress has been exposed via a different ALB w/o problem.

👍 1

freezing-boots-56761

05/30/2023, 4:19 PM

my intuition is that the issue is with this annotation:

Copy code

<http://alb.ingress.kubernetes.io/backend-protocol|alb.ingress.kubernetes.io/backend-protocol>: HTTP

freezing-boots-56761

05/30/2023, 4:20 PM

i added:

<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'

in there for parity, but not sure its necessary since you are only listening on 443

damp-potato-83989

05/30/2023, 4:22 PM

I've tried with and w/o the backend-protocol. It's my understanding from reading that if you don't define it, then it defaults to HTTP. I just re-configured with just those annotations (and some labels we need to add)

damp-potato-83989

05/30/2023, 4:23 PM

Same errors. 😞

freezing-boots-56761

05/30/2023, 4:37 PM

ah yea you're right

damp-potato-83989

05/30/2023, 4:38 PM

I shouldn't have to do https from the alb -> flyte, right? I'll need to if we move forward, but for now just trying to test

freezing-boots-56761

05/30/2023, 4:39 PM

i dont think so

freezing-boots-56761

05/30/2023, 4:39 PM

can you

kubectl get ingress

freezing-boots-56761

05/30/2023, 4:39 PM

are the hostnames different for the 2 ingresses?

damp-potato-83989

05/30/2023, 4:41 PM

Yes

freezing-boots-56761

05/30/2023, 4:42 PM

looking at this: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#ingressgroup implying there are multiple ALBs right now.

damp-potato-83989

05/30/2023, 4:43 PM

An ingress group allows you to share 1 ingress with multiple backend applications that are the same or very, very similar. So, if you wanted to have say 5 flyte instances running, you can share an alb rather than having 5 different albs

freezing-boots-56761

05/30/2023, 4:46 PM

but if you have multiple ALBs, which ALB do you route your host to?

freezing-boots-56761

05/30/2023, 4:47 PM

when you do

kubectl get ingress

are the ALB addresses different?

damp-potato-83989

05/30/2023, 4:48 PM

Right now we have 2 albs. One for the http traffic, and another for grpc. This probably isn't workable pattern for flyte, but atm we haven't gotten there yet. Just trying to get grpc access working through the ingress via grpcurl or grpc_cli. Once we have that it'll be easy to combine the two ingresses into a single one. Unless you think having them separate is causing problems?

freezing-boots-56761

05/30/2023, 4:49 PM

unfortunately, we need to keep them separate in this case because of the GRPC-specific annotations

damp-potato-83989

05/30/2023, 4:49 PM

Yes, right now the two albs (via kubectl get ingress) have different alb addresses

freezing-boots-56761

05/30/2023, 4:49 PM

but i think the group.name will allow them to be merged into a single ALB

damp-potato-83989

05/30/2023, 4:49 PM

Correct

damp-potato-83989

05/30/2023, 4:49 PM

I just haven't done that step yet. 😄

freezing-boots-56761

05/30/2023, 4:50 PM

in the code snippet above:

<hostname>:443

were you using the ALB address or your internal hostname?

damp-potato-83989

05/30/2023, 4:50 PM

I figured I could verify grpc functionality first to try to reduce the potential causes of problems

damp-potato-83989

05/30/2023, 4:51 PM

ALB address. I'm accessing from my laptop, not a pod on the eks cluster.

freezing-boots-56761

05/30/2023, 4:51 PM

i see

damp-potato-83989

05/30/2023, 4:51 PM

I'm using the same kind of url as we use for the http, which successfully gets us to flyte's console

damp-potato-83989

05/30/2023, 4:53 PM

Although, just noticed that ports listed in

kubectl get ingress

for the grpc is port 80. I would think it should be port 443?

damp-potato-83989

05/30/2023, 4:54 PM

% kubectl -n morgan-test get ingress NAME CLASS HOSTS ADDRESS PORTS AGE grpc alb <url> <http://internal-k8s-morgante-morgante-73da0a0e06-1041092043.us-east-2.elb.amazonaws.com|<>internal_url> 80 5d23h http alb <http://flyte-web-east2.api-platform-dev.aws-nonprod.fmrcloud.com|<>url> <http://internal-k8s-morgante-morgante-7641a7c2a3-1671587638.us-east-2.elb.amazonaws.com|<>internal_url> 80 10d

freezing-boots-56761

05/30/2023, 4:59 PM

im seeing the same in my test cluster as well

damp-potato-83989

05/30/2023, 5:09 PM

Same errors? Are you running https from alb->flyte or http?

freezing-boots-56761

05/30/2023, 5:11 PM

alb -> flyte

damp-potato-83989

05/30/2023, 5:12 PM

I mean what protocol between alb->flyte. http or https?

freezing-boots-56761

05/30/2023, 5:14 PM

http

freezing-boots-56761

05/30/2023, 5:14 PM

try this:

Copy code

flytectl get project --admin.endpoint=<hostname>:443

damp-potato-83989

05/30/2023, 5:17 PM

That won't need to use http right? That's a grpc only call?

freezing-boots-56761

05/30/2023, 5:17 PM

yea thats a grpc call

freezing-boots-56761

05/30/2023, 5:17 PM

you can confirm with a port-forward of just the grpc port and using

localhost:8089

👍 1

damp-potato-83989

05/30/2023, 5:20 PM

Well, flytectl seems to have worked? $ flytectl get project --admin.endpoint=<hostname>:443 INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. ------------- ------------- ------------------------- | ID | NAME | DESCRIPTION | ------------- ------------- ------------------------- | flytesnacks | flytesnacks | flytesnacks description | ------------- ------------- ------------------------- 1 rows

freezing-boots-56761

05/30/2023, 5:20 PM

cool

freezing-boots-56761

05/30/2023, 5:20 PM

so your ingress is good

damp-potato-83989

05/30/2023, 5:21 PM

Huh. I guess I didn't think I could use grpcurl on a portforward but not through the ingress

freezing-boots-56761

05/30/2023, 5:22 PM

can you paste your grpcurl command, and output?

freezing-boots-56761

05/30/2023, 5:22 PM

i think it might be because we're not exposing all the grpc methods in the ingress. but not 100% sure.

damp-potato-83989

05/30/2023, 5:22 PM

grpcurl <hostname>:443 describe /flyteidl.service.IdentityService

damp-potato-83989

05/30/2023, 5:22 PM

That endpoint is one that is listed in the ingress definition

freezing-boots-56761

05/30/2023, 5:24 PM

Copy code

server does not support the reflection API

suggests there are reflection specific methods, perhaps?

freezing-boots-56761

05/30/2023, 5:25 PM

see: https://github.com/solo-io/gloo/issues/3244#issue-647618923

freezing-boots-56761

05/30/2023, 5:26 PM

specifcally:

Copy code

grpc.reflection.v1alpha.ServerReflection

damp-potato-83989

05/30/2023, 5:26 PM

I have been looking around on this, and it seems the server needs to do somethings to support reflection. I thought all that was done though since I can do the grpcurl call through the service when it is port-forwarded. So, since I can do an explain over port-forward, I thought the server supported reflection and I should be able to do the same through the ingress

damp-potato-83989

05/30/2023, 5:27 PM

Not a grpc expert at all though. 😄

freezing-boots-56761

05/30/2023, 5:27 PM

right but the ingress doesnt include that route

freezing-boots-56761

05/30/2023, 5:27 PM

ill try and get that working

damp-potato-83989

05/30/2023, 5:27 PM

damp-potato-83989

05/30/2023, 5:28 PM

Reflection is a different route?

freezing-boots-56761

05/30/2023, 5:29 PM

Copy code

> docker run fullstorydev/grpcurl -v -plaintext host.docker.internal:8089 list
flyteidl.service.AdminService
flyteidl.service.AuthMetadataService
flyteidl.service.DataProxyService
flyteidl.service.IdentityService
flyteidl.service.SignalService
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection

damp-potato-83989

05/30/2023, 5:30 PM

Oh, I see. I was seeing that error but didn't realize that was a different route

damp-potato-83989

05/30/2023, 5:30 PM

The information was all in front of me from the start

freezing-boots-56761

05/30/2023, 5:30 PM

we can just expose it in the ingress

👍 1

damp-potato-83989

05/30/2023, 5:35 PM

TY for the help and your patience. :)

👍 1

freezing-boots-56761

05/30/2023, 5:36 PM

hopefully someone will find this useful 🙂

👍 1

damp-potato-83989

05/30/2023, 5:38 PM

One last question while we're on the subject. Are there docs for configuring alb<->flyte access via https?

freezing-boots-56761

05/30/2023, 5:59 PM

sorry idk about docs, but you should be just able to turn on ssl: https://github.com/flyteorg/flyteadmin/blob/15d5c7bb568b570890b1c696f0d01466ea6bef2d/pkg/config/config.go#L23 https://github.com/flyteorg/flyteadmin/blob/15d5c7bb568b570890b1c696f0d01466ea6bef2d/pkg/config/config.go#LL69C37-L69C37

👍 1

1543 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.