I'm getting this error on flytescheduler running on AWS EKS, which I guess is admin ip:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:81: i/o timeout"

panic: authentication error! Original Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken, Auth Error: failed to initialized token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:81: i/o timeout"

Could this be related to auth config? And if so where do I look to fix it?

cc @Prafulla Mahindrakar

hey @Andrew Korzhuev, can you check if the admin service is reachable . The error is i/o timeout so seems the service is not reachable. Also you can get the endpoint which scheduler reaches using

kubectl get cm -n flyte flyscheduler

and check the admin section . That endpoint should be reachable inorder for scheduler to be able to send requests to admin . Check if something is misconfigured here or admin is down.

The endpoint in the config matches flyte-core-grpc ingress, the admin is accessible through UI and CLI otherwise

Does configured client id in the scheduler admin section valid and this should be different client id than the one used by flyteconsole to authenticate. Specifically this section of flyte auth doc https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#id1

I couldn't get this to work until stumbling on that thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1666169843332409

I see. yes there are few gaps seems since we tightened on the security aspects of not having a default secret. Yes if you are using selfAuthServer then you have define those static clients and keep the same values in both places. Usually folks use there Auth server and not the admin provided inmemory auth server.