Still struggling to get this working. flyte-binary...
# flyte-deployment
g
Still struggling to get this working. flyte-binary now deploys and I can access the console via my IdP (i.e.
userAuth
), and I have successfully submitted a workflow (embedded OAuth2 –
appAuth
using defaults, i.e. not explicitly configured), however it stalls at that point producing the following error:
Copy code
E0726 20:02:10.952926       2 workers.go:102] error syncing 'flytesnacks-development/fa2b0208403a24dd2b5a': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:181"},"level":"info","msg":"Processing Workflow.","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:1112","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Node not yet started, will not finalize","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881067","routine":"worker","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a120 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a630 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"fa2b0208403a24dd2b5a\" \u003e producer_id:\"propeller\" phase:FAILED occurred_at:\u003cseconds:1690401730 nanos:953387521 \u003e error:\u003ccode:\"Workflow abort failed\" message:\"Workflow[flytesnacks:development:example.training_workflow] failed. RuntimeExecutionError: max number of system retry attempts [32747/10] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\\nResponse: {\\\"error\\\":\\\"invalid_client\\\",\\\"error_description\\\":\\\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\\\"}]\" kind:SYSTEM \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:351","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:367"},"level":"info","msg":"Completed processing workflow.","ts":"2023-07-26T20:02:10Z"}
The best I can work out for this is that the service (flyte itself) is trying to authenticate to something and failing. What precisely is failing and what it is attempting to authenticate to is not clear.
g
@Samhita Alla thanks again, reading these now. For the record, I did search discuss.flyte.org and GH issues/PRs before I asked on here but clearly not well enough.
After some further attempts it looks like I’ve got enough set up to run the examples. Thanks again for the help.
n
@Greg Linklater how did you solve it? I am facing the same issue with the latest flyte-binary helm chart (1.9.0), using the internal authorization server
below are the configuration values
Copy code
configuration:
  auth:
    enabled: true
    internal:
      clientSecret: foobar
      clientSecretHash: JDJiJDA2JExQRVZBVGVVMWZ3dmNta0ZMWHoxaGVIZ0lqM2NCNzJIVk9rYndrenV6cGxSNmE3WkE1RDRL
    oidc:
      baseUrl: <https://accounts.google.com>
      clientId: <CLIENTID>.<http://apps.googleusercontent.com|apps.googleusercontent.com>
      clientSecret: <CLIENTSECRET>
    authorizedUris:
      - <https://flyte.mydomain>
OIDC works properly but the internal authorization flow which is flytepropeller -> flyteadmin according to this discussion, is broken
@Samhita Alla any suggestions?
s
I hope you looked at the Slack threads. cc @David Espejo (he/him)
n
yeah from the slack thread you shared the key part is to have a random client secret and hash it. The latest docs also includes this. Did I miss anything?
d
@Nan Qin so you mean, the console redirects you to Google but the CLI doesn't?
or what's the behavior?
n
console and cli works fine
The issue is I cannot create an execution, logs from flyte pod:
Copy code
│ flyte E0815 20:01:17.517578       7 workers.go:102] error syncing 'stained-glass/f1a306529979f4e648c3': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized   │││ flyte Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
@David Espejo (he/him) any ideas?
d
@Nan Qin I'm checking and trying to reproduce
n
@David Espejo (he/him) I just found out it was my side
I used kustomize to delete the
config-secret
generated by helm. I have another sealed secret resource for the db password. Didn't realize the
clientSecretHash
was also part of the
config-secret
d
thanks for confirming. Let us know if any other issue arises.
g
@Nan Qin apologies for the silence. Not sure if this is still useful but to anyone else who discovers this thread: Helm:
Copy code
configuration:
  inlineSecretRef: flyte

  auth:
    enabled: true
    clientSecretsExternalSecretRef: flyte-admin-secrets

    oidc:
      baseUrl: <https://keycloak.example.com/realms/flyte>
      clientId: placeholder # This must be here

      authorizedUris:
         - <https://flyte.example.com>
Secret:
Copy code
kind: Secret
metadata:
  name: flyte
apiVersion: v1
stringData:
  204-auth-secrets.yaml: |
    server:
      security:
        useAuth: true

    auth:
      appAuth:
        authServerType: External

        externalAuthServer:
          baseUrl: <https://keycloak.example.com/realms/flyte>
          metadataUrl: .well-known/openid-configuration
          allowedAudience:
            - <https://flyte.example.com>
            - flyte
            - account

        thirdPartyConfig:
          flyteClient:
            clientId: flytectl
            redirectUri: <http://localhost:53593/callback>
            scopes:
              - offline
              - all
              
      userAuth:
        openId:
          baseUrl: <https://keycloak.example.com/realms/flyte>
          scopes:
            - profile
            - openid
            - offline_access
          clientId: flyte
          clientSecretFile: /etc/secrets/oidc_client_secret
I realize it isn’t ideal; but it does work somewhat. I haven’t had a chance to experiment with this env since.