https://flyte.org
Join Slack
Hey all, I'm running flyte with 1.14.0 `flyte-binary` and trying to run a workflow developed with `f...
# flyte-support
g
Hey all, I'm running flyte with 1.14.0 
flyte-binary
and trying to run a workflow developed with 
flytekit==1.13.6
Currently setting up a new staging environment with a new cluster. Getting the following error in the logs of 
flyte-binary
though, as the workflow is not being scheduled:
Copy code
"Failed to record workflow event [execution_id:{project:"flytesnacks" domain:"development" name:"alvkslsgjkfnnl5v2ptf"} producer_id:"propeller" phase:RUNNING occurred_at:{seconds:1740478696 nanos:672294747}] with err: authentication error! Original Error: <nil>, Auth Error: failed to issue token. Error: failed to get new token: failed to get new token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).""
What could it be? Currently checking the role/bindings of my GCP service account that is attached to the K8s service account
Downgrading back to 1.13.2 on 
flyte-binary
doesn't make a difference, so I'm assuming I'm missing some config related to auth
Found a bit more errors messages. I see that it's getting a 503 from NGINX (the ingress controller I'm using), which then is reporting a 401 when doing the following:
Copy code
"POST /oauth2/token HTTP/2.0" 401 174 "-" "Go-http-client/2.0" 100 0.006 [flyte-flyte-backend-flyte-binary-http-8088]
Not sure how to proceed
Seems like flytepropeller is trying to get an Oauth2 token to be able to schedule and generate the pods for the workflow tasks, but it keeps failing for some reason?
Going to try a fresh install and reset the db to see if the issue goes away like that.
Mind you that registering and starting a workflow execution via client (e.g. FlyteRemote) does work
However somehow internal communication seems to fail
Fixed!
The issue was the internal client secret I had set, it had some special characters that were causing troubles.
I assume then that the secret would be used, and the computed hashes would not match up with the clientSecretHash I set in the deployment, so yeah. The 401/unauthorized response makes sense then
a
@gentle-tomato-480 hey thanks for sharing, glad you fixed it
g
@average-finland-92144 Yeah, happy I did. The error messages were pretty vague. Thankfully other people had already found it as well. https://discuss.flyte.org/t/3156427/hey-everyone-slightly-smiling-face-i-m-currently-deploying-a and https://discuss.flyte.org/t/13387442/still-struggling-to-get-this-working-flyte-binary-now-deploy And then it clicked that it probably had to be the client secret not matching the hash
7 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by