Hello everyone. We’re deploying flyte-binary on EK...
# flyte-deploymentm
Mike Morgan
05/30/2023, 2:20 PMHello everyone. We’re deploying flyte-binary on EKS. Pod is running and connecting to the database. I am able to load the dashboard and trying to run a simple workflow, but I am getting the following error
Copy code
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
details: failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed
caused by: Post <https://sts.us-east-2.amazonaws.com/>: dial tcp 52.95.18.19:443: i/o timeout
Debug string UNKNOWN:Error received from peer ipv4:192.168.65.2:8081 {created_time:"2023-05-30T12:31:18.554380596+00:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \<https://sts.us-east-2.amazonaws.com/\>: dial tcp 52.95.18.19:443: i/o timeout"}Mike Morgan
05/30/2023, 2:21 PMOther pods on the same eks cluster, namespace and using the same service account are able to use awscli
y
Yee
05/30/2023, 5:18 PM
can you try creating a link manually?
Yee
05/30/2023, 5:18 PM
may need to spin up a awscli/python pod with the same credentials as flyte-binary.
Yee
05/30/2023, 5:18 PM
haven’t seen this before. timeout is strange.
m
Mike Morgan
05/30/2023, 5:24 PMYes, indeed. Let me try something
Mike Morgan
05/30/2023, 5:37 PMWhat did you mean by creating a link manually?
n
Neil Haria
05/30/2023, 6:16 PMIt's probably because your RDS and EKS are not in the same vpc
m
Mike Morgan
05/30/2023, 6:28 PMThey actually are, but can you elaborate on your thought process?
y
Yee
05/30/2023, 6:47 PM
like make a pod with awscli
Yee
05/30/2023, 6:47 PM
and ssh into it, and call
aws get-signed-urlm
Mike Morgan
05/31/2023, 2:05 AMI am able to create a pre-signed url with AWS cli using the same service account, but still getting the same time out error above when calling pyflyte run
y
Yee
05/31/2023, 4:27 PM
the timeout is coming from the admin side.
Yee
05/31/2023, 4:30 PM
this is pretty weird. i’m not sure. this feels like just a networking issue. the go code in admin isn’t doing anything special.
Yee
05/31/2023, 4:30 PM
if configured incorrectly it should get a 403 or a 401 or something, not a timeout
Yee
05/31/2023, 4:31 PM
this feels like the flyte/admin pod isn’t able to talk to sts.
151 Views