Still struggling to get this working. flyte-binary now deploys and I can access the console via my IdP (i.e.

userAuth

), and I have successfully submitted a workflow (embedded OAuth2 –

appAuth

using defaults, i.e. not explicitly configured), however it stalls at that point producing the following error:

E0726 20:02:10.952926       2 workers.go:102] error syncing 'flytesnacks-development/fa2b0208403a24dd2b5a': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:181"},"level":"info","msg":"Processing Workflow.","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:1112","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Node not yet started, will not finalize","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881067","routine":"worker","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a120 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a630 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"fa2b0208403a24dd2b5a\" \u003e producer_id:\"propeller\" phase:FAILED occurred_at:\u003cseconds:1690401730 nanos:953387521 \u003e error:\u003ccode:\"Workflow abort failed\" message:\"Workflow[flytesnacks:development:example.training_workflow] failed. RuntimeExecutionError: max number of system retry attempts [32747/10] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\\nResponse: {\\\"error\\\":\\\"invalid_client\\\",\\\"error_description\\\":\\\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\\\"}]\" kind:SYSTEM \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:351","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:367"},"level":"info","msg":"Completed processing workflow.","ts":"2023-07-26T20:02:10Z"}

The best I can work out for this is that the service (flyte itself) is trying to authenticate to something and failing. What precisely is failing and what it is attempting to authenticate to is not clear.

Here are a couple of references for you: • https://discuss.flyte.org/t/3156427/hey-everyone-slightly-smiling-face-i-m-currently-deploying-a • https://github.com/flyteorg/flyte/pull/3062 • https://github.com/flyteorg/flyte/pull/3837/files

@Samhita Alla thanks again, reading these now. For the record, I did search discuss.flyte.org and GH issues/PRs before I asked on here but clearly not well enough.

@Greg Linklater how did you solve it? I am facing the same issue with the latest flyte-binary helm chart (1.9.0), using the internal authorization server

I hope you looked at the Slack threads. cc @David Espejo (he/him)

yeah from the slack thread you shared the key part is to have a random client secret and hash it. The latest docs also includes this. Did I miss anything?

@Nan Qin so you mean, the console redirects you to Google but the CLI doesn't?

console and cli works fine

@Nan Qin I'm checking and trying to reproduce

@David Espejo (he/him) I just found out it was my side

thanks for confirming. Let us know if any other issue arises.

@Nan Qin apologies for the silence. Not sure if this is still useful but to anyone else who discovers this thread: Helm:

configuration:
  inlineSecretRef: flyte

  auth:
    enabled: true
    clientSecretsExternalSecretRef: flyte-admin-secrets

    oidc:
      baseUrl: <https://keycloak.example.com/realms/flyte>
      clientId: placeholder # This must be here

      authorizedUris:
         - <https://flyte.example.com>

Secret:

kind: Secret
metadata:
  name: flyte
apiVersion: v1
stringData:
  204-auth-secrets.yaml: |
    server:
      security:
        useAuth: true

    auth:
      appAuth:
        authServerType: External

        externalAuthServer:
          baseUrl: <https://keycloak.example.com/realms/flyte>
          metadataUrl: .well-known/openid-configuration
          allowedAudience:
            - <https://flyte.example.com>
            - flyte
            - account

        thirdPartyConfig:
          flyteClient:
            clientId: flytectl
            redirectUri: <http://localhost:53593/callback>
            scopes:
              - offline
              - all
              
      userAuth:
        openId:
          baseUrl: <https://keycloak.example.com/realms/flyte>
          scopes:
            - profile
            - openid
            - offline_access
          clientId: flyte
          clientSecretFile: /etc/secrets/oidc_client_secret

I realize it isn’t ideal; but it does work somewhat. I haven’t had a chance to experiment with this env since.