Still struggling to get this working flyte binary now deploy

Question

Still struggling to get this working flyte binary now deploys and I can access the console via my IdP i e `userAuth` and I have successfully submitted a workflow embedded OAuth2 `appAuth` using defaults i e not explicitly configured however it stalls at that point producing the following error ```E0726 20 02 10 952926 2 workers go 102 error syncing flytesnacks development fa2b0208403a24dd2b5a Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error failed to get token oauth2 cannot fetch token 401 Unauthorized Response error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method json src controller go 159 level info msg == u003e Enqueueing workflow flytesnacks development fa2b0208403a24dd2b5a ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development routine worker 8 src handler go 181 level info msg Processing Workflow ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development res ver 188881069 routine worker 8 src executor go 1112 wf flytesnacks development example training workflow level info msg Node not yet started will not finalize ts 2023 07 26T20 02 10Z json src token go 37 level info msg Error occurred in NewAccessRequest invalid client ts 2023 07 26T20 02 10Z json src token go 37 level info msg Error occurred in NewAccessRequest invalid client ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development res ver 188881067 routine worker src token source provider go 230 wf flytesnacks development example training workflow level warning msg failed to get token w oauth2 RetrieveError= u0026 0xc23d93a120 123 34 34 125 ts 2023 07 26T20 02 10Z json src token go 37 level info msg Error occurred in NewAccessRequest invalid client ts 2023 07 26T20 02 10Z json src token go 37 level info msg Error occurred in NewAccessRequest invalid client ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development res ver 188881069 routine worker 8 src token source provider go 230 wf flytesnacks development example training workflow level warning msg failed to get token w oauth2 RetrieveError= u0026 0xc23d93a630 123 34 34 125 ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development res ver 188881069 routine worker 8 src workflow event recorder go 69 wf flytesnacks development example training workflow level info msg Failed to record workflow event execution id u003cproject flytesnacks domain development name fa2b0208403a24dd2b5a u003e producer id propeller phase FAILED occurred at u003cseconds 1690401730 nanos 953387521 u003e error u003ccode Workflow abort failed message Workflow flytesnacks development example training workflow failed RuntimeExecutionError max number of system retry attempts 32747 10 exhausted Last known status message Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error failed to get token oauth2 cannot fetch token 401 Unauthorized nResponse error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method kind SYSTEM u003e with err EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error failed to get token oauth2 cannot fetch token 401 Unauthorized nResponse error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development res ver 188881069 routine worker 8 src executor go 351 wf flytesnacks development example training workflow level warning msg Event recording failed Error EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error failed to get token oauth2 cannot fetch token 401 Unauthorized nResponse error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method ts 2023 07 26T20 02 10Z json src controller go 159 level info msg == u003e Enqueueing workflow flytesnacks development fa2b0208403a24dd2b5a ts 2023 07 26T20 02 10Z json exec id fa2b0208403a24dd2b5a ns flytesnacks development routine worker 8 src handler go 367 level info msg Completed processing workflow ts 2023 07 26T20 02 10Z ``` The best I can work out for this is that the service flyte itself is trying to authenticate to something and failing What precisely is failing and what it is attempting to authenticate to is not clear

Samhita Alla · Answer

Here are a couple of references for you

Greg Linklater · Answer

< Samhita Alla> thanks again reading these now For the record I did search and GH issues PRs before I asked on here but clearly not well enough

Greg Linklater · Answer

After some further attempts it looks like I ve got enough set up to run the examples Thanks again for the help

Nan Qin · Answer

< Greg Linklater> how did you solve it I am facing the same issue with the latest flyte binary helm chart 1 9 0 using the internal authorization server

Nan Qin · Answer

below are the configuration values ```configuration auth enabled true internal clientSecret foobar clientSecretHash JDJiJDA2JExQRVZBVGVVMWZ3dmNta0ZMWHoxaGVIZ0lqM2NCNzJIVk9rYndrenV6cGxSNmE3WkE1RDRL oidc baseUrl clientId lt CLIENTID gt clientSecret lt CLIENTSECRET gt authorizedUris ```

Nan Qin · Answer

OIDC works properly but the internal authorization flow which is flytepropeller gt flyteadmin according to <https discuss flyte org t 3156427 hey everyone slightly smiling face i m currently deploying a|this discussion> is broken

Nan Qin · Answer

< Samhita Alla> any suggestions

Samhita Alla · Answer

I hope you looked at the Slack threads cc < David Espejo he him >

Nan Qin · Answer

yeah from the slack thread you shared the key part is to have a random client secret and hash it The <https docs flyte org en latest deployment configuration auth setup html apply oidc configuration|latest docs> also includes this Did I miss anything

David Espejo (he/him) · Answer

< Nan Qin> so you mean the console redirects you to Google but the CLI doesn t

David Espejo (he/him) · Answer

or what s the behavior

Nan Qin · Answer

console and cli works fine

Nan Qin · Answer

The issue is I cannot create an execution logs from flyte pod ``` flyte E0815 20 01 17 517578 7 workers go 102 error syncing stained glass f1a306529979f4e648c3 Workflow failed ErrorRecordingError failed to publish event caused by EventSinkError Error sending event caused by rpc error code = Unauthenticated desc = transport per RPC creds failed due to error failed to get token oauth2 cannot fetch token 401 Unauthorized flyte Response error invalid client error description Client authentication failed e g unknown client no client authentication included or unsupported authentication method ```

Nan Qin · Answer

< David Espejo he him > any ideas

David Espejo (he/him) · Answer

< Nan Qin> I m checking and trying to reproduce

Nan Qin · Answer

< David Espejo he him > I just found out it was my side

Nan Qin · Answer

I used kustomize to delete the `config secret` generated by helm I have another sealed secret resource for the db password Didn t realize the `clientSecretHash` was also part of the `config secret`

David Espejo (he/him) · Answer

thanks for confirming Let us know if any other issue arises

Greg Linklater · Answer

< Nan Qin> apologies for the silence Not sure if this is still useful but to anyone else who discovers this thread Helm ```configuration inlineSecretRef flyte auth enabled true clientSecretsExternalSecretRef flyte admin secrets oidc baseUrl clientId placeholder This must be here authorizedUris ``` Secret ```kind Secret metadata name flyte apiVersion v1 stringData 204 auth secrets yaml | server security useAuth true auth appAuth authServerType External externalAuthServer baseUrl metadataUrl well known openid configuration allowedAudience flyte account thirdPartyConfig flyteClient clientId flytectl redirectUri scopes offline all userAuth openId baseUrl scopes profile openid offline access clientId flyte clientSecretFile etc secrets oidc client secret``` I realize it isn t ideal but it does work somewhat I haven t had a chance to experiment with this env since