https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
r

Rob Rati

05/30/2023, 3:19 PM
I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by following this: https://docs.flyte.org/en/latest/deployment/deployment/cloud_production.html And we have the http endpoint exposed via alb w/o issue. We use the aws ingress controller to create the ALBs, so I've modified the grpc ingress definition to add the following annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP alb.ingress.kubernetes.io/backend-protocol-version: GRPC alb.ingress.kubernetes.io/certificate-arn: <redacted> alb.ingress.kubernetes.io/healthcheck-path: /grpc.health.v1.Health alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: EKS-ALB-sg alb.ingress.kubernetes.io/target-type: ip And the spec with: ingressClassName: alb rules: - host: <http://flyte-web-east2-grpc.api-platform-dev.aws-nonprod.fmrcloud.com|<>hostname> But whenever I try to use grpcurl to access the endpoint to list/describe I always get: $ grpcurl <hostname>:443 list Failed to list services: server does not support the reflection API $ grpcurl <hostname>:443 describe /flyteidl.service.IdentityService Failed to resolve symbol "/flyteidl.service.IdentityService": server does not support the reflection API When I try to use grpc_cli ls <hostname>:443 I get an error: Trying to connect an http1.x server I can describe the grpc endpoints when I port-forward the grpc service, so the server is working as expected. Has anyone else tried to do this?
d

David Espejo (he/him)

05/30/2023, 3:57 PM
Hi @Rob Rati In this guide, there are annotations that work with the most recent 
flyte-binary
chart version using ALB: https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/08-adjust-values-upgrade-Helm.md
j

jeev

05/30/2023, 3:58 PM
GRPC needs http2 that only works over SSL (443) on ALB afaik.
r

Rob Rati

05/30/2023, 3:58 PM
I have ssl to the ALB, but atm it's http/unsecured from the ALB to flyte
j

jeev

05/30/2023, 4:00 PM
i believe you also need: alb.ingress.kubernetes.io/backend-protocol-version: GRPC as an annotation on the GPRC-only ingress
r

Rob Rati

05/30/2023, 4:01 PM
I have that
I also think I have all the annotations in that "flyte the hard way" doc. Slight difference in that the scheme is internal instead of internet-facing, but otherwise I think I have (or have tried) all the options in that doc
Unless the group.name is important? That's for sharing an ingress afaik, and we only have 1 flyte instance we are working on atm
j

jeev

05/30/2023, 4:17 PM
can you try ONLY the following annotations:
Copy code
<http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: '<ARN>'
<http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTPS":443}]'
<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internal
<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
<http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
for just the GRPC ingress? the HTTP endpoints should be in a different ingress
r

Rob Rati

05/30/2023, 4:18 PM
Sure, yas. For clarity, Http endpoints are in a different ingress and that ingress has been exposed via a different ALB w/o problem.
j

jeev

05/30/2023, 4:19 PM
my intuition is that the issue is with this annotation:
Copy code
<http://alb.ingress.kubernetes.io/backend-protocol|alb.ingress.kubernetes.io/backend-protocol>: HTTP
i added: 
<http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
in there for parity, but not sure its necessary since you are only listening on 443
r

Rob Rati

05/30/2023, 4:22 PM
I've tried with and w/o the backend-protocol. It's my understanding from reading that if you don't define it, then it defaults to HTTP. I just re-configured with just those annotations (and some labels we need to add)
Same errors. 😞
j

jeev

05/30/2023, 4:37 PM
ah yea you're right
r

Rob Rati

05/30/2023, 4:38 PM
I shouldn't have to do https from the alb -> flyte, right? I'll need to if we move forward, but for now just trying to test
j

jeev

05/30/2023, 4:39 PM
i dont think so
can you 
kubectl get ingress
are the hostnames different for the 2 ingresses?
r

Rob Rati

05/30/2023, 4:41 PM
Yes
j

jeev

05/30/2023, 4:42 PM
r

Rob Rati

05/30/2023, 4:43 PM
An ingress group allows you to share 1 ingress with multiple backend applications that are the same or very, very similar. So, if you wanted to have say 5 flyte instances running, you can share an alb rather than having 5 different albs
j

jeev

05/30/2023, 4:46 PM
but if you have multiple ALBs, which ALB do you route your host to?
when you do 
kubectl get ingress
are the ALB addresses different?
r

Rob Rati

05/30/2023, 4:48 PM
Right now we have 2 albs. One for the http traffic, and another for grpc. This probably isn't workable pattern for flyte, but atm we haven't gotten there yet. Just trying to get grpc access working through the ingress via grpcurl or grpc_cli. Once we have that it'll be easy to combine the two ingresses into a single one. Unless you think having them separate is causing problems?
j

jeev

05/30/2023, 4:49 PM
unfortunately, we need to keep them separate in this case because of the GRPC-specific annotations
r

Rob Rati

05/30/2023, 4:49 PM
Yes, right now the two albs (via kubectl get ingress) have different alb addresses
j

jeev

05/30/2023, 4:49 PM
but i think the group.name will allow them to be merged into a single ALB
r

Rob Rati

05/30/2023, 4:49 PM
Correct
I just haven't done that step yet. 😄
j

jeev

05/30/2023, 4:50 PM
in the code snippet above: 
<hostname>:443
were you using the ALB address or your internal hostname?
r

Rob Rati

05/30/2023, 4:50 PM
I figured I could verify grpc functionality first to try to reduce the potential causes of problems
ALB address. I'm accessing from my laptop, not a pod on the eks cluster.
j

jeev

05/30/2023, 4:51 PM
i see
r

Rob Rati

05/30/2023, 4:51 PM
I'm using the same kind of url as we use for the http, which successfully gets us to flyte's console
Although, just noticed that ports listed in 
kubectl get ingress
for the grpc is port 80. I would think it should be port 443?
j

jeev

05/30/2023, 4:59 PM
im seeing the same in my test cluster as well
r

Rob Rati

05/30/2023, 5:09 PM
Same errors? Are you running https from alb->flyte or http?
j

jeev

05/30/2023, 5:11 PM
alb -> flyte
r

Rob Rati

05/30/2023, 5:12 PM
I mean what protocol between alb->flyte. http or https?
j

jeev

05/30/2023, 5:14 PM
http
try this:
Copy code
flytectl get project --admin.endpoint=<hostname>:443
r

Rob Rati

05/30/2023, 5:17 PM
That won't need to use http right? That's a grpc only call?
j

jeev

05/30/2023, 5:17 PM
yea thats a grpc call
you can confirm with a port-forward of just the grpc port and using 
localhost:8089
r

Rob Rati

05/30/2023, 5:20 PM
Well, flytectl seems to have worked? $ flytectl get project --admin.endpoint=<hostname>:443 INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. ------------- ------------- ------------------------- | ID | NAME | DESCRIPTION | ------------- ------------- ------------------------- | flytesnacks | flytesnacks | flytesnacks description | ------------- ------------- ------------------------- 1 rows
j

jeev

05/30/2023, 5:20 PM
cool
so your ingress is good
r

Rob Rati

05/30/2023, 5:21 PM
Huh. I guess I didn't think I could use grpcurl on a portforward but not through the ingress
j

jeev

05/30/2023, 5:22 PM
can you paste your grpcurl command, and output?
i think it might be because we're not exposing all the grpc methods in the ingress. but not 100% sure.
r

Rob Rati

05/30/2023, 5:22 PM
grpcurl <hostname>:443 describe /flyteidl.service.IdentityService
That endpoint is one that is listed in the ingress definition
j

jeev

05/30/2023, 5:24 PM
Copy code
server does not support the reflection API
suggests there are reflection specific methods, perhaps?
specifcally:
Copy code
grpc.reflection.v1alpha.ServerReflection
r

Rob Rati

05/30/2023, 5:26 PM
I have been looking around on this, and it seems the server needs to do somethings to support reflection. I thought all that was done though since I can do the grpcurl call through the service when it is port-forwarded. So, since I can do an explain over port-forward, I thought the server supported reflection and I should be able to do the same through the ingress
Not a grpc expert at all though. 😄
j

jeev

05/30/2023, 5:27 PM
right but the ingress doesnt include that route
ill try and get that working
r

Rob Rati

05/30/2023, 5:27 PM
Oh
Reflection is a different route?
j

jeev

05/30/2023, 5:29 PM
Copy code
> docker run fullstorydev/grpcurl -v -plaintext host.docker.internal:8089 list
flyteidl.service.AdminService
flyteidl.service.AuthMetadataService
flyteidl.service.DataProxyService
flyteidl.service.IdentityService
flyteidl.service.SignalService
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
r

Rob Rati

05/30/2023, 5:30 PM
Oh, I see. I was seeing that error but didn't realize that was a different route
The information was all in front of me from the start
j

jeev

05/30/2023, 5:30 PM
we can just expose it in the ingress
r

Rob Rati

05/30/2023, 5:35 PM
TY for the help and your patience. :)
j

jeev

05/30/2023, 5:36 PM
hopefully someone will find this useful 🙂
r

Rob Rati

05/30/2023, 5:38 PM
One last question while we're on the subject. Are there docs for configuring alb<->flyte access via https?
j

jeev

05/30/2023, 5:59 PM
21 Views
Powered by