https://flyte.org logo
Powered by
#flyte-deployment
Title
# flyte-deployment
m

Mike Morgan

05/30/2023, 2:20 PM
Hello everyone. We’re deploying flyte-binary on EKS. Pod is running and connecting to the database. I am able to load the dashboard and trying to run a simple workflow, but I am getting the following error
Copy code
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
        details: failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed
caused by: Post <https://sts.us-east-2.amazonaws.com/>: dial tcp 52.95.18.19:443: i/o timeout
        Debug string UNKNOWN:Error received from peer ipv4:192.168.65.2:8081 {created_time:"2023-05-30T12:31:18.554380596+00:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \<https://sts.us-east-2.amazonaws.com/\>: dial tcp 52.95.18.19:443: i/o timeout"}
Other pods on the same eks cluster, namespace and using the same service account are able to use awscli
y

Yee

05/30/2023, 5:18 PM
can you try creating a link manually?
may need to spin up a awscli/python pod with the same credentials as flyte-binary.
haven’t seen this before. timeout is strange.
m

Mike Morgan

05/30/2023, 5:24 PM
Yes, indeed. Let me try something
What did you mean by creating a link manually?
n

Neil Haria

05/30/2023, 6:16 PM
It's probably because your RDS and EKS are not in the same vpc
m

Mike Morgan

05/30/2023, 6:28 PM
They actually are, but can you elaborate on your thought process?
y

Yee

05/30/2023, 6:47 PM
like make a pod with awscli
and ssh into it, and call 
aws get-signed-url
m

Mike Morgan

05/31/2023, 2:05 AM
I am able to create a pre-signed url with AWS cli using the same service account, but still getting the same time out error above when calling pyflyte run
y

Yee

05/31/2023, 4:27 PM
the timeout is coming from the admin side.
this is pretty weird. i’m not sure. this feels like just a networking issue. the go code in admin isn’t doing anything special.
if configured incorrectly it should get a 403 or a 401 or something, not a timeout
this feels like the flyte/admin pod isn’t able to talk to sts.
2 Views
Powered by