I need help in configuring flyte with Keycloak I ...
# announcements
s
I need help in configuring flyte with Keycloak I have setup 2 clients, flytectl(access type public) and flytepropeller(access type confidential with client Id and secret). I am following the keycloak section in the https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html# My relevant values in the helm values file are as below secrets: adminOauthClientCredentials: # -- If enabled is true, helm will create and manage
flyte-secret-auth
and populate it with
clientSecret
. # If enabled is false, it's up to the user to create
flyte-secret-auth
as described in # https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server enabled: true clientSecret: "CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C" clientId: "flytepropeller" -- FlyteAdmin server configuration adminServer: # Refer to the server config. server: httpPort: 8088 grpcPort: 8089 security: # -- Controls whether to serve requests over SSL/TLS. secure: false # -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. useAuth: true allowCors: true allowedOrigins: # Accepting all domains for Sandbox installation - "*" allowedHeaders: - "Content-Type" - "flyte-authorization" # Refer to the full structure for documentation. flyteadmin: roleNameKey: "iam.amazonaws.com/role" profilerPort: 10254 metricsScope: "flyte:" metadataStoragePrefix: - "metadata" - "admin" eventVersion: 2 testing: host: http://flyteadmin # -- Authentication configuration auth: authorizedUris: # This should point at your public http Uri. - https://flytedeployment url # This will be used by internal services in the same namespace as flyteadmin - http://flyteadmin:80 # This will be used by internal services in the same cluster but different namespaces - http://flyteadmin.nmlp.svc.cluster.local:80
Copy code
# Controls app authentication config
  appAuth:
    thirdPartyConfig:
      flyteClient:
        clientId: flytectl
        redirectUri: https://<flyte deployment url>/callback
        scopes:
          - offline
          - all

  # Controls user authentication
  userAuth:
    openId:
      baseUrl: https://<keycloak production realm>/realms/nsdmlp
      scopes:
        - profile
        - openid
      clientId: flytepropeller
I have also edited the flyte-admin-secrets to have the correct client secret apiVersion: v1 data: claim_symmetric_key: cWlBYzlYWHdLN3lnaksrWUJGdStFUlRYK0RDdlk4SjVjZFJtaXBTcDBhdw== cookie_block_key: ejZPdkhrZ1crWXdib21JZHdVZ05IOGJESVp0OE5KWnNZT285KzIyRVM1dw== cookie_hash_key: Q093TUY2RTdOMW5MeFZ4Rnk1dGNzZGN5NU5aeTNWN2JTMXRPTjBLUGhQQ2JWZ3hGby9XQkVRdi84Yjk4ZEIyeEV3Zm5KYURDVzFkSjBuSGZrbS8zYVE= oidc_client_secret: CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C With all the above setting, when i try to login to the flyte console via the ingress, it gives me the below message in the flyteadmin pod logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T183411Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T183412Z"} Logs from 8/5/2022, 120239 AM And the chrome errors out with 403 access denied page. I have tried secret and id with and without quotes(just in case if helm was acting crazy) but it still gives the same error. Please assist. Thanks, Sujith
Cc @Neal Feierabend / @Sören Brunk
s
@Neal Feierabend @Sören Brunk @Ketan (kumare3) I have set up 3 clients in keycloak, nsdaiml(user client), flytectl and flytepropeller. flytectl is setup without authentication. nsdaiml and flytepropeller are setup with cred. I have also set up offline and all client scopes in keycloak. I also added aud to the access token by creating a client mapper and put allowedaudience as flyteclient in the helm values. After doing these settings also I am unable to get the access token and i get the below errors in the flyteadmin logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T183411Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T183412Z"}
Should the flytepropeller client be used for userauth??
s
We had an extra client for flyteconsole auth like
nsdaiml
in your case. That's the one to put into userAuth:
Copy code
userAuth:
            openId:
              baseUrl: https://<keycloak-domain>/auth/realms/<realm>
              clientId: nsdaiml
And we've put the secret of that client into
flyte-admin-secrets
as described here: https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#apply-configuration
s
@Ketan (kumare3), is there any way to enable debug in the flyteadmin container so that I can see what exactly is happening in the login execution
any flag which I can pass to the container to get debug logs in the pod
I enabled everything as per @Sören Brunk but still getting the login error as earlier
s
@Sujith Samuel can you try to restart the flyteadmin pod after setting the secret, just to be sure it picks up the right one?
s
Tried that @Sören Brunk> Still the same error. {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-05T131647Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"unauthorized_client\",\"error_description\":\"Invalid client secret\"}","ts":"2022-08-05T131712Z"}
s
You're trying to login via flyteconsole right?
s
Yes, via the ingress
s
If you execute
kubectl get secrets/flyte-admin-secrets --template='{{.data.oidc_client_secret | base64decode}}'
do you get the right keycloak secret of your client back?
s
kubectl get secrets/flyte-admin-secrets -n nmlp --template='{{.data.oidc_client_secret | base64decode }}' I▒▒▒L▒F▒▒▒J▒▒t▒g▒▒▒} <- This is not right samuel@samuel-vm-001:~/repos/flyteadmin$ kubectl get secrets/flyte-admin-secrets -n nmlp -o yaml apiVersion: v1 data: oidc_client_secret: SeL8npmV9UyBRvC3x9FKgoV0u2e6yvR9 <- This is right
s
@Sujith Samuel are you sure you've set it via
stringData
as described in the docs? Like this:
Copy code
stringData:
  oidc_client_secret: <client_secret from the previous step>
data:
  ...
This will ensure the expected base64 encoding
s
@Sören Brunk, oops that was a very very rookie mistake from my side. Apologies but I put it under data and not stringdata
Now its working great
👍 1
I am able to reach the console
Thanks a lot @Sören Brunk for you time and assistance
s
Glad I could help. I think the keycloak docs can still be improved. So if you've noticed something missing/confusing reading them, please don't hesitate to open an issue. Or even better a PR if you'd like to contribute an improvement 🙂
s
I saw that Ketan had sent another issue about Keycloak docs https://github.com/flyteorg/flyte/issues/2606 Are those changes valid. About adding the aud mapper inside clients?
s
This is essentially what we did a while ago to create the client/mapper for flytepropeller via script. Not sure if it is still all necessary:
Copy code
FLYTEPROPELLER_CLIENT_ID=$(kcadm create clients -i -r ${REALM} \
            -s clientId="flytepropeller" \
            -s redirectUris="[\"https://<flyte-domain>/*\"]" \
            -s webOrigins="[\"+\"]" \
            -s serviceAccountsEnabled="true" \
            -s defaultClientScopes="[\"all\", \"offline\"]" \
            -s publicClient="false")
echo creating flytepropeller audience mapper
kcadm create clients/${FLYTEPROPELLER_CLIENT_ID}/protocol-mappers/models -r ${REALM} \
  -s name=audience-mapper \
  -s protocol=openid-connect \
  -s protocolMapper=oidc-audience-mapper \
  -s config.\"included.custom.audience\"="https://<flyte-domain>" \
  -s config.\"access.token.claim\"="true" \
  -s config.\"id.token.claim\"="false"
echo creating flytepropeller scp mapper
kcadm create clients/${FLYTEPROPELLER_CLIENT_ID}/protocol-mappers/models -r ${REALM} \
  -b '{"name":"scp-claim-mapper","protocol":"openid-connect","protocolMapper":"oidc-hardcoded-claim-mapper","consentRequired":false,"config":{"claim.value":"[\"all\"]","userinfo.token.claim":"false","id.token.claim":"false","access.token.claim":"true","claim.name":"scp","jsonType.label":"JSON","access.tokenResponse.claim":"false"}}'
similar with the flytectl client
s
@Sören Brunk Need your help again. This time with the flytectl config file I am using the below details admin: endpoint: dns:///flyte.test.aimlframe.he-test.k8s.dyn.nesc.nokia.net authorizationServerUrl: https://keycloakrealm/realms/nsdmlp tokenUrl: https://keycloak.realm/realms/nsdmlp/protocol/openid-connect/token caCertFilePath: /etc/ssl/certs/ca-certificates.crt insecure: false # Update auth type to
Pkce
or
ClientSecret
authType: ClientSecret # Set to the clientId (will be used for both Pkce and ClientSecret flows) # Leave empty to use the value discovered through flyteAdmin's Auth discovery endpoint. clientId: nsdmlclient # Set to the location where the client secret is mounted. # Only needed/used for
ClientSecret
flow. clientSecretLocation: /home/samuel/install/flyte/clientsecret I am getting the below error while running the flytectl version command samuel@samuel-vm-001:~/.flyte$ flytectl version {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-08-05T210624+03:00"} {"json":{"src":"client.go:183"},"level":"error","msg":"failed to initialize token source provider. Err: rpc error: code = Unknown desc = : HTTP status code 0; transport: missing content-type field","ts":"2022-08-05T210624+03:00"} {"json":{"src":"client.go:188"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-08-05T210624+03:00"} {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-08-05T210624+03:00"} { "App": "flytectl", "Build": "2b13e14", "Version": "0.6.5", "BuildTime": "2022-08-05 210624.178889411 +0300 EEST m=+0.022552240" }{"json":{"src":"version.go:103"},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unknown desc = : HTTP status code 0; transport: missing content-type field: \n","ts":"2022-08-05T210624+03:00"} {"json":{"src":"version.go:81"},"level":"debug","msg":"rpc error: code = Unknown desc = : HTTP status code 0; transport: missing content-type field","ts":"2022-08-05T210624+03:00"}
i resolved the above error. I had not create a separate GRPC ingress for flytectl to work. I create it and now its able to connect to the upstream but there is a 500 INTERNAL SERVER ERROR. Would it be possible to send a sample flytectl config file which you have used @Sören Brunk @Ketan (kumare3)
Now the error is as in below samuel@samuel-vm-001:~/.flyte$ flytectl version {"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-08-05T223855+03:00"} {"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-08-05T223855+03:00"} {"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-08-05T223855+03:00"} {"json":{},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-08-05T223855+03:00"} {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-08-05T223856+03:00"} { "App": "flytectl", "Build": "2b13e14", "Version": "0.6.5", "BuildTime": "2022-08-05 223855.880583512 +0300 EEST m=+0.021980189" }{"json":{},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 500 Internal Server Error\nResponse: {\"error\":\"unknown_error\"}: \n","ts":"2022-08-05T223856+03:00"} {"json":{},"level":"debug","msg":"rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 500 Internal Server Error\nResponse: {\"error\":\"unknown_error\"}","ts":"2022-08-05T223856+03:00"} @Ketan (kumare3) Kindly let me know how to enable more debug information in the flyteadmin pod.
s
@Sujith Samuel my .flyte/config.yaml looks like this:
Copy code
admin:
  endpoint: dns:///<flyte-uri>
  authType: Pkce
logger:
  show-source: true
  level: 3
flytectl is configured as a public client in Keycloak. flytectl will use the browser for oauth.
k
hey @Sujith Samuel maybe we can jump on a call and help on Monday or something
s
@Sören Brunk i am trying to use flytectl inside a container to submit workflows to k8s cluster. i cant get a browser opened inside this Is there any way to use flytectl inside a container with keycloak enabled ?
Is there any way I can set auth header inside flytectl command.. if there is any example for getting a token from keycloak and then setting it in the header, that would be awesome
k
Yes there is a way
You need to generate client secrets and use that
Let me share the link
But you can simply use the api in that case to submit workflows, I would use Flyte remote
You can add FLYTE_CREDENTIALS_CLIENT_SECRET to the lambda’s environment variables as part of initializing FlyteRemote. To do so: Follow steps 1-3 outlined in the Lambda ARN section. Choose Configuration and then choose Environment Variables. Set the key as FLYTE_CREDENTIALS_CLIENT_SECRET, and the value should be your secret.
s
@Ketan (kumare3), I am now trying to use external auth for userauth and selfauth for appauth.
# Controls app authentication config appAuth: authServerType: Self externalAuthServer: allowedAudience: [] baseUrl: "" metadataUrl: "" # 2. Optional: Set external auth server baseUrl if different from OpenId baseUrl. selfAuthServer: accessTokenLifespan: 30m0s authorizationCodeLifespan: 5m0s claimSymmetricEncryptionKeySecretName: claim_symmetric_key issuer: "" oldTokenSigningRSAKeySecretName: token_rsa_key_old.pem refreshTokenLifespan: 1h0m0s staticClients: flyte-cli: audience: null grant_types: - refresh_token - authorization_code id: flyte-cli public: true redirect_uris: - http://localhost:53593/callback - http://localhost:12345/callback response_types: - code - token scopes: - all - offline - access_token flytectl: audience: null grant_types: - refresh_token - authorization_code id: flytectl public: true redirect_uris: - http://localhost:53593/callback - http://localhost:12345/callback response_types: - code - token scopes: - all - offline - access_token flytepropeller: audience: null client_secret: JDJhJDA2JHB4czFBa0c4MUt2cmhwbWwxUWlMU09RYVRrOWVlUHJVLzdZYWI5eTA3aDN4MFRnbGJhb1Q2 grant_types: - all - offline - refresh_token - client_credentials id: flytepropeller public: false redirect_uris: - http://localhost:3846/callback response_types: - token scopes: - all - offline - access_token tokenSigningRSAKeySecretName: token_rsa_key.pem thirdPartyConfig: flyteClient: #audience: null clientId: flytectl redirectUri: http://localhost:53593/callback scopes: - all - offline # Controls user authentication userAuth: redirectUrl: "https://flyteredirect/console" cookieSetting: domain: "flytedomain" sameSitePolicy: DefaultMode openId: baseUrl: "https://keycloakrealm/realms/nsdmlp" scopes: - profile - openid - offline_access clientId: nsdmlclient clientSecretName: oidc_client_secret
In the flytectl logs I see that its trying to access http://localhost:53593/callback But I dont see this port service running on the pod. Does this need to be enabled via any other config
On windows, flytectl is working fine but through Linux or a container I am not able to get this same setup working
s
Is there any way to use flytectl inside a container with keycloak enabled ?
@Sujith Samuel we actually had a similar setup. For that case we had an additional keycloak client
flytectl-internal
which was a confidential client with
Service Accounts Enabled
(aka client credentials grant in oauth terms). I'm trying to find the corresponding flytectl config.
s
That would be awesome if you could provide that sample....
and the flyteadmin config values for auth section, user and app
s
I'm sorry @Sujith Samuel I had recalled that wrongly. We were using our own registration via GRPC against flyteadmin, not flytectl.
s
Right. @Sören Brunk
@Ketan (kumare3), If possible could you please provide me a sample command which I can put in the command section of the flytectl config file to get the token for the public client. I am planning to use a public client to get its token using an external command which could possibly bypass opening the browser for token in the unix container
Copy code
--admin.command strings                      Command for external authentication token generation
k
Cc @Babis Kiosidis ? Cc @Prafulla Mahindrakar
b
Hey, here is an example
Copy code
cat ~/.flyte/config.yaml
 admin:
   endpoint: myflyteadmin.host:443
   insecure: false
   authType: ExternalCommand
   command: [gcloud,auth,print-identity-token]
p
In case you want to use client_credentials , this would be kind of config you would need and which wont require browser access.
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.org
  # Change insecure flag to ensure that you use the right setting for your environment
  insecure: false
  clientId: *********
  authType: ClientSecret
  clientSecretLocation: /home/runner/secret_location
logger:
  # Logger settings to control logger output. Useful to debug logger:
  show-source: true
  level: 4
where in
/home/runner/secret_location
contains the client_secret
s
I am using clientsecret but I get invalid client error for flytepropeller even though I have defined it in keycloak
p
this i suggested for flytectl issue that you were running into since you don’t want browser based auth .
s
flytectl is defined as public client. For its secret, which value should I use, the token???? since there is no secret for public client
p
You can reuse the propeller one .
s
@Babis Kiosidis I want to extract token using curl to keycloak and jq command curl -X POST "https://kctokenendpoint/token" -H "Content-Type: application/x-www-form-urlencoded" -d "username=***" -d "password=***" -d 'grant_type=password' -d "client_id=flytectl" | jq -r '.access_token' How can this be given there. Please assist.
@Prafulla Mahindrakar When i use the cliensecret mode for flytectl I get the below error }{"json":{"src":"version.go:103"},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client. The OAuth 2.0 Client is marked as public and is thus not allowed to use authorization grant 'client_credentials'.\"}: \n","ts":"2022-08-08T192415+05:30"} {"json":{"src":"version.go:81"},"level":"debug","msg":"rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client. The OAuth 2.0 Client is marked as public and is thus not allowed to use authorization grant 'client_credentials'.\"}","ts":"2022-08-08T192415+05:30"}
p
whats the client_id your are using .
s
flytectl
p
can you share your confgi.yaml
can you use propeller one or create a clone of it and use it.
s
so that means that I have to mention flytepropeller in the third party config for appauth right????
thirdPartyConfig: flyteClient: #audience: null clientId: flytectl redirectUri: http://localhost:53593/callback scopes: - all - offline This is my current setting and wit flytectl being a public ID, it requires a web browser
p
yes you can ypdate it here and also in your config.yaml for flytectl use that clientId
s
do the clientids for app auth have to be flytectl, flytepropeller or flyte-cli only or they can be anything else too. For userauth I am using something which is none of the above??
p
they should match whats in the admins configmap
s
thanks. And does this client need to have any mappers like audience or groups set in keycloak???
p
I think you would need something similar to what this user did for the mappers https://github.com/flyteorg/flyte/issues/2606
s
Now i get this error }{"json":{"src":"version.go:103"},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}: \n","ts":"2022-08-08T171948+03:00"} {"json":{"src":"version.go:81"},"level":"debug","msg":"rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}","ts":"2022-08-08T171948+03:00"}
p
Can we get on a call that would be easier
s
That would be good. What time would you prefer please
p
Can we sync in around 9 pm IST if you are available . Or we can do morning anything after 10 am IST
s
would 9 PM be ok for you today. If fine, I can send an invite you you
p
works for me .
s
👍 1
Changed the 3rd party config to thirdPartyConfig: flyteClient: audience: flytepropeller clientId: flytepropeller redirectUri: http://localhost:53593/callback scopes: - all - offline - access_token and added flytepropeller as audience in the mapper section for flytepropeller client and now I am able to access flytectl without browser
Thanks a lot for all your help @Prafulla Mahindrakar @Sören Brunk @Ketan (kumare3)
k
cc @Sujith Samuel can you please recommend what we should update in the docs
s
I think for the on prem installation of kubernetes, you could add a new section which outlines some extra things like setting up the mappers in flytepropeller, setting up the well-known config in external authserver section et al.
in fact i sent my values file yesterday to Prafulla. I can clean it up and send it so that it can be documented in your docs website
🙏 1
k
Please make a PR, the community will love it
👍 1
s
Opened PR 2746
❤️ 3
719 Views