I need help in configuring flyte with Keycloak I have setup 2 clients, flytectl(access type public) and flytepropeller(access type confidential with client Id and secret). I am following the keycloak section in the https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html# My relevant values in the helm values file are as below secrets: adminOauthClientCredentials: # -- If enabled is true, helm will create and manage

flyte-secret-auth

and populate it with

clientSecret

. # If enabled is false, it's up to the user to create

flyte-secret-auth

as described in # https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server enabled: true clientSecret: "CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C" clientId: "flytepropeller" -- FlyteAdmin server configuration adminServer: # Refer to the server config. server: httpPort: 8088 grpcPort: 8089 security: # -- Controls whether to serve requests over SSL/TLS. secure: false # -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. useAuth: true allowCors: true allowedOrigins: # Accepting all domains for Sandbox installation - "*" allowedHeaders: - "Content-Type" - "flyte-authorization" # Refer to the full structure for documentation. flyteadmin: roleNameKey: "iam.amazonaws.com/role" profilerPort: 10254 metricsScope: "flyte:" metadataStoragePrefix: - "metadata" - "admin" eventVersion: 2 testing: host: http://flyteadmin # -- Authentication configuration auth: authorizedUris: # This should point at your public http Uri. - https://flytedeployment url # This will be used by internal services in the same namespace as flyteadmin - http://flyteadmin:80 # This will be used by internal services in the same cluster but different namespaces - http://flyteadmin.nmlp.svc.cluster.local:80

# Controls app authentication config
  appAuth:
    thirdPartyConfig:
      flyteClient:
        clientId: flytectl
        redirectUri: https://<flyte deployment url>/callback
        scopes:
          - offline
          - all

  # Controls user authentication
  userAuth:
    openId:
      baseUrl: https://<keycloak production realm>/realms/nsdmlp
      scopes:
        - profile
        - openid
      clientId: flytepropeller

I have also edited the flyte-admin-secrets to have the correct client secret apiVersion: v1 data: claim_symmetric_key: cWlBYzlYWHdLN3lnaksrWUJGdStFUlRYK0RDdlk4SjVjZFJtaXBTcDBhdw== cookie_block_key: ejZPdkhrZ1crWXdib21JZHdVZ05IOGJESVp0OE5KWnNZT285KzIyRVM1dw== cookie_hash_key: Q093TUY2RTdOMW5MeFZ4Rnk1dGNzZGN5NU5aeTNWN2JTMXRPTjBLUGhQQ2JWZ3hGby9XQkVRdi84Yjk4ZEIyeEV3Zm5KYURDVzFkSjBuSGZrbS8zYVE= oidc_client_secret: CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C With all the above setting, when i try to login to the flyte console via the ingress, it gives me the below message in the flyteadmin pod logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T183411Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T183412Z"} Logs from 8/5/2022, 120239 AM And the chrome errors out with 403 access denied page. I have tried secret and id with and without quotes(just in case if helm was acting crazy) but it still gives the same error. Please assist. Thanks, Sujith

@Neal Feierabend @Sören Brunk @Ketan (kumare3) I have set up 3 clients in keycloak, nsdaiml(user client), flytectl and flytepropeller. flytectl is setup without authentication. nsdaiml and flytepropeller are setup with cred. I have also set up offline and all client scopes in keycloak. I also added aud to the access token by creating a client mapper and put allowedaudience as flyteclient in the helm values. After doing these settings also I am unable to get the access token and i get the below errors in the flyteadmin logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T183411Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T183412Z"}

We had an extra client for flyteconsole auth like

nsdaiml

in your case. That's the one to put into userAuth:

userAuth:
            openId:
              baseUrl: https://<keycloak-domain>/auth/realms/<realm>
              clientId: nsdaiml

@Ketan (kumare3), is there any way to enable debug in the flyteadmin container so that I can see what exactly is happening in the login execution

@Sujith Samuel can you try to restart the flyteadmin pod after setting the secret, just to be sure it picks up the right one?

Tried that @Sören Brunk> Still the same error. {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-05T131647Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"unauthorized_client\",\"error_description\":\"Invalid client secret\"}","ts":"2022-08-05T131712Z"}

You're trying to login via flyteconsole right?

Yes, via the ingress

If you execute

kubectl get secrets/flyte-admin-secrets --template='{{.data.oidc_client_secret | base64decode}}'

do you get the right keycloak secret of your client back?

kubectl get secrets/flyte-admin-secrets -n nmlp --template='{{.data.oidc_client_secret | base64decode }}' I▒▒▒L▒F▒▒▒J▒▒t▒g▒▒▒} <- This is not right samuel@samuel-vm-001:~/repos/flyteadmin$ kubectl get secrets/flyte-admin-secrets -n nmlp -o yaml apiVersion: v1 data: oidc_client_secret: SeL8npmV9UyBRvC3x9FKgoV0u2e6yvR9 <- This is right

@Sujith Samuel are you sure you've set it via

stringData

as described in the docs? Like this:

stringData:
  oidc_client_secret: <client_secret from the previous step>
data:
  ...

@Sören Brunk, oops that was a very very rookie mistake from my side. Apologies but I put it under data and not stringdata

Glad I could help. I think the keycloak docs can still be improved. So if you've noticed something missing/confusing reading them, please don't hesitate to open an issue. Or even better a PR if you'd like to contribute an improvement 🙂

I saw that Ketan had sent another issue about Keycloak docs https://github.com/flyteorg/flyte/issues/2606 Are those changes valid. About adding the aud mapper inside clients?

This is essentially what we did a while ago to create the client/mapper for flytepropeller via script. Not sure if it is still all necessary:

FLYTEPROPELLER_CLIENT_ID=$(kcadm create clients -i -r ${REALM} \
            -s clientId="flytepropeller" \
            -s redirectUris="[\"https://<flyte-domain>/*\"]" \
            -s webOrigins="[\"+\"]" \
            -s serviceAccountsEnabled="true" \
            -s defaultClientScopes="[\"all\", \"offline\"]" \
            -s publicClient="false")
echo creating flytepropeller audience mapper
kcadm create clients/${FLYTEPROPELLER_CLIENT_ID}/protocol-mappers/models -r ${REALM} \
  -s name=audience-mapper \
  -s protocol=openid-connect \
  -s protocolMapper=oidc-audience-mapper \
  -s config.\"included.custom.audience\"="https://<flyte-domain>" \
  -s config.\"access.token.claim\"="true" \
  -s config.\"id.token.claim\"="false"
echo creating flytepropeller scp mapper
kcadm create clients/${FLYTEPROPELLER_CLIENT_ID}/protocol-mappers/models -r ${REALM} \
  -b '{"name":"scp-claim-mapper","protocol":"openid-connect","protocolMapper":"oidc-hardcoded-claim-mapper","consentRequired":false,"config":{"claim.value":"[\"all\"]","userinfo.token.claim":"false","id.token.claim":"false","access.token.claim":"true","claim.name":"scp","jsonType.label":"JSON","access.tokenResponse.claim":"false"}}'

@Sören Brunk Need your help again. This time with the flytectl config file I am using the below details admin: endpoint: dns:///flyte.test.aimlframe.he-test.k8s.dyn.nesc.nokia.net authorizationServerUrl: https://keycloakrealm/realms/nsdmlp tokenUrl: https://keycloak.realm/realms/nsdmlp/protocol/openid-connect/token caCertFilePath: /etc/ssl/certs/ca-certificates.crt insecure: false # Update auth type to

Pkce

or

ClientSecret

authType: ClientSecret # Set to the clientId (will be used for both Pkce and ClientSecret flows) # Leave empty to use the value discovered through flyteAdmin's Auth discovery endpoint. clientId: nsdmlclient # Set to the location where the client secret is mounted. # Only needed/used for

ClientSecret

flow. clientSecretLocation: /home/samuel/install/flyte/clientsecret I am getting the below error while running the flytectl version command samuel@samuel-vm-001:~/.flyte$ flytectl version {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-08-05T210624+03:00"} {"json":{"src":"client.go:183"},"level":"error","msg":"failed to initialize token source provider. Err: rpc error: code = Unknown desc = : HTTP status code 0; transport: missing content-type field","ts":"2022-08-05T210624+03:00"} {"json":{"src":"client.go:188"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-08-05T210624+03:00"} {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-08-05T210624+03:00"} { "App": "flytectl", "Build": "2b13e14", "Version": "0.6.5", "BuildTime": "2022-08-05 210624.178889411 +0300 EEST m=+0.022552240" }{"json":{"src":"version.go:103"},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unknown desc = : HTTP status code 0; transport: missing content-type field: \n","ts":"2022-08-05T210624+03:00"} {"json":{"src":"version.go:81"},"level":"debug","msg":"rpc error: code = Unknown desc = : HTTP status code 0; transport: missing content-type field","ts":"2022-08-05T210624+03:00"}

@Sujith Samuel my .flyte/config.yaml looks like this:

admin:
  endpoint: dns:///<flyte-uri>
  authType: Pkce
logger:
  show-source: true
  level: 3

flytectl is configured as a public client in Keycloak. flytectl will use the browser for oauth.

hey @Sujith Samuel maybe we can jump on a call and help on Monday or something

@Sören Brunk i am trying to use flytectl inside a container to submit workflows to k8s cluster. i cant get a browser opened inside this Is there any way to use flytectl inside a container with keycloak enabled ?

Yes there is a way

@Ketan (kumare3), I am now trying to use external auth for userauth and selfauth for appauth.

Is there any way to use flytectl inside a container with keycloak enabled ?

@Sujith Samuel we actually had a similar setup. For that case we had an additional keycloak client

flytectl-internal

which was a confidential client with

Service Accounts Enabled

(aka client credentials grant in oauth terms). I'm trying to find the corresponding flytectl config.

That would be awesome if you could provide that sample....

I'm sorry @Sujith Samuel I had recalled that wrongly. We were using our own registration via GRPC against flyteadmin, not flytectl.

Right. @Sören Brunk

Cc @Babis Kiosidis ? Cc @Prafulla Mahindrakar

Hey, here is an example

cat ~/.flyte/config.yaml
 admin:
   endpoint: myflyteadmin.host:443
   insecure: false
   authType: ExternalCommand
   command: [gcloud,auth,print-identity-token]

In case you want to use client_credentials , this would be kind of config you would need and which wont require browser access.

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.org
  # Change insecure flag to ensure that you use the right setting for your environment
  insecure: false
  clientId: *********
  authType: ClientSecret
  clientSecretLocation: /home/runner/secret_location
logger:
  # Logger settings to control logger output. Useful to debug logger:
  show-source: true
  level: 4

where in

/home/runner/secret_location

contains the client_secret

I am using clientsecret but I get invalid client error for flytepropeller even though I have defined it in keycloak

this i suggested for flytectl issue that you were running into since you don’t want browser based auth .

flytectl is defined as public client. For its secret, which value should I use, the token???? since there is no secret for public client

You can reuse the propeller one .

@Babis Kiosidis I want to extract token using curl to keycloak and jq command curl -X POST "https://kctokenendpoint/token" -H "Content-Type: application/x-www-form-urlencoded" -d "username=***" -d "password=***" -d 'grant_type=password' -d "client_id=flytectl" | jq -r '.access_token' How can this be given there. Please assist.

whats the client_id your are using .

flytectl

can you share your confgi.yaml

so that means that I have to mention flytepropeller in the third party config for appauth right????

yes you can ypdate it here and also in your config.yaml for flytectl use that clientId

do the clientids for app auth have to be flytectl, flytepropeller or flyte-cli only or they can be anything else too. For userauth I am using something which is none of the above??

they should match whats in the admins configmap

thanks. And does this client need to have any mappers like audience or groups set in keycloak???

I think you would need something similar to what this user did for the mappers https://github.com/flyteorg/flyte/issues/2606

Now i get this error }{"json":{"src":"version.go:103"},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}: \n","ts":"2022-08-08T171948+03:00"} {"json":{"src":"version.go:81"},"level":"debug","msg":"rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}","ts":"2022-08-08T171948+03:00"}

Can we get on a call that would be easier

That would be good. What time would you prefer please

Can we sync in around 9 pm IST if you are available . Or we can do morning anything after 10 am IST

would 9 PM be ok for you today. If fine, I can send an invite you you

works for me .

https://meet.google.com/qjp-cvff-fay Kindly join at 9 PM

cc @Sujith Samuel can you please recommend what we should update in the docs

I think for the on prem installation of kubernetes, you could add a new section which outlines some extra things like setting up the mappers in flytepropeller, setting up the well-known config in external authserver section et al.

Please make a PR, the community will love it

Opened PR 2746