https://flyte.org logo
#ask-the-community
Title
# ask-the-community
l

Lukas Bommes

08/28/2023, 10:35 AM
Dear all, I am trying to enable SSO with keycloak in Flyte. I followed the tutorial in the docs (https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html) and got everything working described here. So, basically auth works fine in the UI, flytectl and also flytekit.remote. However, when starting a workflow, the workflow status remains in "UNKNOWN" and the flyte-binary logs report lots of these messages
Copy code
E0828 10:23:23.869876       7 workers.go:102] error syncing 'my-namespace/a7clnsk5kgb5x6shkf8p': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://my-url.com>:port/oauth2/token": dial tcp: lookup <http://my-url.com|my-url.com> on 10.96.0.10:53: no such host]
and
Copy code
E0828 12:48:56.147140       7 workers.go:102] error syncing 'ohli-core-development/avhv2n5js764zfdjjj4j': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<https://my-url.com>:port/oauth2/token": x509: certificate signed by unknown authority]
Furthermore, I observed that etcd is being written to at a very high rate (5 MB per second), which takes down the whole cluster after a short while as etcd ecxeeds is storage quota and starts failing. I am using flyte-binary v1.9.0 and these are my current helm values
Copy code
configuration:

  database:
    ...

  storage:
    ...

  inline:
    admin:
      additionalVolumes:
      - name: ca-pemstore
        configMap:
          name: ca-pemstore
      additionalVolumeMounts:
      - name: ca-pemstore
        mountPath: /etc/ssl/certs/my_root_ca.crt
        subPath: my_root_ca.crt
        readOnly: false
  
  auth:
    # auth docs: <https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html>
    enabled: true
    oidc:
      baseUrl: <https://auth.my-url.com:3334/realms/flyte>
      clientId: flyte
      clientSecret: ***
    internal:
      clientSecret: '***'
      clientSecretHash: ***
    authorizedUris:
    - <https://my-url.com>:port
    - <http://flyte-backend.flyte:8088>
    - <https://my-url.com>:port/oauth2/token

ingress:
  create: true
  host: <http://my-url.com|my-url.com>
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    # needed to fix: <https://stackoverflow.com/questions/56126864/why-do-i-get-502-when-trying-to-authenticate>
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "128k"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
As you can see I tried to provide the root CA as described here: https://flyte-org.slack.com/archives/CP2HDHKE1/p1689254783702299?thread_ts=1689247106.327409&amp;cid=CP2HDHKE1 However, this has no effect. Any help would be greatly appreciated.
d

David Espejo (he/him)

08/28/2023, 4:01 PM
@Lukas Bommes I typically see the
ssl-redirect
annotation under
commonAnnotations
instead of
grpcAnnotations
Not sure if that's the cause but it looks different
also, I guess
my-url
is an anonymized version of your URL which is DNS resolvable, right?
l

Lukas Bommes

08/29/2023, 6:09 AM
@David Espejo (he/him) Thanks for the hint on the`ssl-redirect`annotation. It was not the cause for the issue but I will still move it to
commonAnnotations
. And yes, my-url.com is just a placeholder for the actual URL. I did get everything to work now. Let me share my solution for others and for future reference: For the flyte-binary Helm chart I had to add the
deployment.extraVolumes
and `deployment.extraVolumeMounts`as shown below.
ca-pemstore
is a ConfigMap holding the root CA and can be created with:
Copy code
kubectl -n flyte create configmap ca-pemstore --from-file=<path/to/root_ca.crt>
Here, it does not matter whether the root CA is stored as pem or crt file. values.yaml
Copy code
configuration:

  database:
    ...

  storage:
    ...
  
  auth:
    # auth docs: <https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html>
    enabled: true
    oidc:
      baseUrl: <https://auth.my-url.com:3334/realms/flyte>
      clientId: flyte
      clientSecret: ***
    internal:
      clientSecret: '***'
      clientSecretHash: ***
    authorizedUris:
    - <https://my-url.com>:port
    - <http://flyte-backend.flyte:8088>
    - <https://my-url.com>:port/oauth2/token
	
deployment:
  extraVolumes:
  - name: ca-pemstore
    configMap:
      name: ca-pemstore
  extraVolumeMounts:
  - name: ca-pemstore
    mountPath: /etc/ssl/certs/my_cert.crt
    subPath: my_cert.crt
    readOnly: false

ingress:
  create: true
  host: <http://my-url.com|my-url.com>
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    # needed to fix: <https://stackoverflow.com/questions/56126864/why-do-i-get-502-when-trying-to-authenticate>
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "128k"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
Another little issue that I encountered along the way: The baseUrl for Keycloak mentioned in the docs (https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html#apply-oidc-configuration) is not
https://<keycloak-url>/auth/realms/<keycloak-realm>
but
https://<keycloak-url>/realms/<keycloak-realm>
. I guess, this was recently changed in Keycloak.
d

David Espejo (he/him)

08/29/2023, 4:12 PM
Thank you @Lukas Bommes! Would you mind submitting a PR to the docs? Or I can create it and add you as co-author
l

Lukas Bommes

08/30/2023, 1:58 PM
Would be great if you could do create the PR since I have too much other stuff on my plate right now.
13 Views