I m getting this error on flytescheduler running on AWS EKS

Question

I m getting this error on flytescheduler running on AWS EKS which I guess is admin ip ```Error rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 81 i o timeout ``` ```panic authentication error Original Error rpc error code = Unauthenticated desc = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken Auth Error failed to initialized token source provider Err failed to fetch auth metadata Error rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 81 i o timeout ``` Could this be related to auth config And if so where do I look to fix it

Kevin Su · Answer

cc < Prafulla Mahindrakar>

Prafulla Mahindrakar · Answer

hey < Andrew Korzhuev> can you check if the admin service is reachable The error is i o timeout so seems the service is not reachable Also you can get the endpoint which scheduler reaches using `kubectl get cm n flyte flyscheduler` and check the admin section That endpoint should be reachable inorder for scheduler to be able to send requests to admin Check if something is misconfigured here or admin is down

Andrew Korzhuev · Answer

The endpoint in the config matches flyte core grpc ingress the admin is accessible through UI and CLI otherwise

Andrew Korzhuev · Answer

Aha this was the firewall rules between EKS nodes which tripped me

Andrew Korzhuev · Answer

Now I m getting ```panic rpc error code = Unauthenticated desc = transport per RPC creds failed due to error oauth2 cannot fetch token 401 Unauthorized Response error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method ```

Prafulla Mahindrakar · Answer

Does configured client id in the scheduler admin section valid and this should be different client id than the one used by flyteconsole to authenticate Specifically this section of flyte auth doc <https docs flyte org en latest deployment cluster config auth setup html id1>

Prafulla Mahindrakar · Answer

check for ```adminOauthClientCredentials``` if this is configured for your helm charts

Andrew Korzhuev · Answer

I couldn t get this to work until stumbling on that thread <https flyte org slack com archives CP2HDHKE1 p1666169843332409>

Andrew Korzhuev · Answer

Looks like `adminOauthClientCredentials clientSecret` also needs to match `appAuth selfAuthServer` section which is not documented

Prafulla Mahindrakar · Answer

I see yes there are few gaps seems since we tightened on the security aspects of not having a default secret Yes if you are using selfAuthServer then you have define those static clients and keep the same values in both places Usually folks use there Auth server and not the admin provided inmemory auth server

Prafulla Mahindrakar · Answer

Created a doc update ticket for it <https github com flyteorg flyte issues 3029>