Dear all, I am trying to enable SSO with keycloak in Flyte. I followed the tutorial in the docs (https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html) and got everything working described here. So, basically auth works fine in the UI, flytectl and also flytekit.remote. However, when starting a workflow, the workflow status remains in "UNKNOWN" and the flyte-binary logs report lots of these messages

E0828 10:23:23.869876       7 workers.go:102] error syncing 'my-namespace/a7clnsk5kgb5x6shkf8p': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://my-url.com>:port/oauth2/token": dial tcp: lookup <http://my-url.com|my-url.com> on 10.96.0.10:53: no such host]

and

E0828 12:48:56.147140       7 workers.go:102] error syncing 'ohli-core-development/avhv2n5js764zfdjjj4j': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://my-url.com>:port/oauth2/token": x509: certificate signed by unknown authority]

Furthermore, I observed that etcd is being written to at a very high rate (5 MB per second), which takes down the whole cluster after a short while as etcd ecxeeds is storage quota and starts failing. I am using flyte-binary v1.9.0 and these are my current helm values

configuration:

  database:
    ...

  storage:
    ...

  inline:
    admin:
      additionalVolumes:
      - name: ca-pemstore
        configMap:
          name: ca-pemstore
      additionalVolumeMounts:
      - name: ca-pemstore
        mountPath: /etc/ssl/certs/my_root_ca.crt
        subPath: my_root_ca.crt
        readOnly: false
  
  auth:
    # auth docs: <https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html>
    enabled: true
    oidc:
      baseUrl: <https://auth.my-url.com:3334/realms/flyte>
      clientId: flyte
      clientSecret: ***
    internal:
      clientSecret: '***'
      clientSecretHash: ***
    authorizedUris:
    - <https://my-url.com>:port
    - <http://flyte-backend.flyte:8088>
    - <https://my-url.com>:port/oauth2/token

ingress:
  create: true
  host: <http://my-url.com|my-url.com>
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    # needed to fix: <https://stackoverflow.com/questions/56126864/why-do-i-get-502-when-trying-to-authenticate>
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "128k"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC

As you can see I tried to provide the root CA as described here: https://flyte-org.slack.com/archives/CP2HDHKE1/p1689254783702299?thread_ts=1689247106.327409&cid=CP2HDHKE1 However, this has no effect. Any help would be greatly appreciated.

@freezing-shampoo-67249 I typically see the

ssl-redirect

annotation under

commonAnnotations

instead of

grpcAnnotations

Not sure if that's the cause but it looks different

@average-finland-92144 Thanks for the hint on the`ssl-redirect`annotation. It was not the cause for the issue but I will still move it to

commonAnnotations

. And yes, my-url.com is just a placeholder for the actual URL. I did get everything to work now. Let me share my solution for others and for future reference: For the flyte-binary Helm chart I had to add the

deployment.extraVolumes

and `deployment.extraVolumeMounts`as shown below.

ca-pemstore

is a ConfigMap holding the root CA and can be created with:

kubectl -n flyte create configmap ca-pemstore --from-file=<path/to/root_ca.crt>

Here, it does not matter whether the root CA is stored as pem or crt file. values.yaml

configuration:

  database:
    ...

  storage:
    ...
  
  auth:
    # auth docs: <https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html>
    enabled: true
    oidc:
      baseUrl: <https://auth.my-url.com:3334/realms/flyte>
      clientId: flyte
      clientSecret: ***
    internal:
      clientSecret: '***'
      clientSecretHash: ***
    authorizedUris:
    - <https://my-url.com>:port
    - <http://flyte-backend.flyte:8088>
    - <https://my-url.com>:port/oauth2/token
	
deployment:
  extraVolumes:
  - name: ca-pemstore
    configMap:
      name: ca-pemstore
  extraVolumeMounts:
  - name: ca-pemstore
    mountPath: /etc/ssl/certs/my_cert.crt
    subPath: my_cert.crt
    readOnly: false

ingress:
  create: true
  host: <http://my-url.com|my-url.com>
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    # needed to fix: <https://stackoverflow.com/questions/56126864/why-do-i-get-502-when-trying-to-authenticate>
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "128k"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC

Thank you @freezing-shampoo-67249! Would you mind submitting a PR to the docs? Or I can create it and add you as co-author

Would be great if you could do create the PR since I have too much other stuff on my plate right now.