https://flyte.org
Join Slack
I'm getting this error on flytescheduler running on AWS EKS, which I guess is admin ip: ```Error: rp...
# flyte-support
h
I'm getting this error on flytescheduler running on AWS EKS, which I guess is admin ip:
Copy code
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:81: i/o timeout"
Copy code
panic: authentication error! Original Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken, Auth Error: failed to initialized token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:81: i/o timeout"
Could this be related to auth config? And if so where do I look to fix it?
g
cc @icy-agent-73298
i
hey @helpful-kilobyte-23008, can you check if the admin service is reachable . The error is i/o timeout so seems the service is not reachable. Also you can get the endpoint which scheduler reaches using 
kubectl get cm -n flyte flyscheduler
and check the admin section . That endpoint should be reachable inorder for scheduler to be able to send requests to admin . Check if something is misconfigured here or admin is down.
h
The endpoint in the config matches flyte-core-grpc ingress, the admin is accessible through UI and CLI otherwise
Aha, this was the firewall rules between EKS nodes which tripped me
Now I'm getting:
Copy code
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}
i
Does configured client id in the scheduler admin section valid and this should be different client id than the one used by flyteconsole to authenticate. Specifically this section of flyte auth doc https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#id1
check for
Copy code
adminOauthClientCredentials
if this is configured for your helm charts
h
I couldn't get this to work until stumbling on that thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1666169843332409
Looks like 
adminOauthClientCredentials.clientSecret
also needs to match 
appAuth.selfAuthServer
section, which is not documented
i
I see. yes there are few gaps seems since we tightened on the security aspects of not having a default secret. Yes if you are using selfAuthServer then you have define those static clients and keep the same values in both places. Usually folks use there Auth server and not the admin provided inmemory auth server.
Created a doc update ticket for it https://github.com/flyteorg/flyte/issues/3029
394 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by