sparse-pizza-79993
07/26/2023, 8:07 PMuserAuth
), and I have successfully submitted a workflow (embedded OAuth2 – appAuth
using defaults, i.e. not explicitly configured), however it stalls at that point producing the following error:
E0726 20:02:10.952926 2 workers.go:102] error syncing 'flytesnacks-development/fa2b0208403a24dd2b5a': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:181"},"level":"info","msg":"Processing Workflow.","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:1112","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Node not yet started, will not finalize","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881067","routine":"worker","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a120 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a630 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"fa2b0208403a24dd2b5a\" \u003e producer_id:\"propeller\" phase:FAILED occurred_at:\u003cseconds:1690401730 nanos:953387521 \u003e error:\u003ccode:\"Workflow abort failed\" message:\"Workflow[flytesnacks:development:example.training_workflow] failed. RuntimeExecutionError: max number of system retry attempts [32747/10] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\\nResponse: {\\\"error\\\":\\\"invalid_client\\\",\\\"error_description\\\":\\\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\\\"}]\" kind:SYSTEM \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:351","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:367"},"level":"info","msg":"Completed processing workflow.","ts":"2023-07-26T20:02:10Z"}
The best I can work out for this is that the service (flyte itself) is trying to authenticate to something and failing. What precisely is failing and what it is attempting to authenticate to is not clear.tall-lock-23197
sparse-pizza-79993
07/27/2023, 7:21 AMsparse-pizza-79993
07/27/2023, 11:39 AMshy-accountant-549
08/15/2023, 8:10 PMshy-accountant-549
08/15/2023, 9:39 PMconfiguration:
auth:
enabled: true
internal:
clientSecret: foobar
clientSecretHash: JDJiJDA2JExQRVZBVGVVMWZ3dmNta0ZMWHoxaGVIZ0lqM2NCNzJIVk9rYndrenV6cGxSNmE3WkE1RDRL
oidc:
baseUrl: <https://accounts.google.com>
clientId: <CLIENTID>.<http://apps.googleusercontent.com|apps.googleusercontent.com>
clientSecret: <CLIENTSECRET>
authorizedUris:
- <https://flyte.mydomain>
shy-accountant-549
08/15/2023, 9:44 PMshy-accountant-549
08/15/2023, 9:44 PMtall-lock-23197
shy-accountant-549
08/16/2023, 2:39 PMaverage-finland-92144
08/16/2023, 3:17 PMaverage-finland-92144
08/16/2023, 3:18 PMshy-accountant-549
08/16/2023, 3:18 PMshy-accountant-549
08/16/2023, 3:19 PM│ flyte E0815 20:01:17.517578 7 workers.go:102] error syncing 'stained-glass/f1a306529979f4e648c3': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized │││ flyte Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
shy-accountant-549
08/16/2023, 8:56 PMaverage-finland-92144
08/17/2023, 5:33 PMshy-accountant-549
08/17/2023, 5:39 PMshy-accountant-549
08/17/2023, 5:41 PMconfig-secret
generated by helm. I have another sealed secret resource for the db password. Didn't realize the clientSecretHash
was also part of the config-secret
average-finland-92144
08/17/2023, 5:52 PMsparse-pizza-79993
09/03/2023, 7:32 AMconfiguration:
inlineSecretRef: flyte
auth:
enabled: true
clientSecretsExternalSecretRef: flyte-admin-secrets
oidc:
baseUrl: <https://keycloak.example.com/realms/flyte>
clientId: placeholder # This must be here
authorizedUris:
- <https://flyte.example.com>
Secret:
kind: Secret
metadata:
name: flyte
apiVersion: v1
stringData:
204-auth-secrets.yaml: |
server:
security:
useAuth: true
auth:
appAuth:
authServerType: External
externalAuthServer:
baseUrl: <https://keycloak.example.com/realms/flyte>
metadataUrl: .well-known/openid-configuration
allowedAudience:
- <https://flyte.example.com>
- flyte
- account
thirdPartyConfig:
flyteClient:
clientId: flytectl
redirectUri: <http://localhost:53593/callback>
scopes:
- offline
- all
userAuth:
openId:
baseUrl: <https://keycloak.example.com/realms/flyte>
scopes:
- profile
- openid
- offline_access
clientId: flyte
clientSecretFile: /etc/secrets/oidc_client_secret
I realize it isn’t ideal; but it does work somewhat. I haven’t had a chance to experiment with this env since.