https://flyte.org logo
Join Slack
Powered by
<@U06RKAMK0UC> During PKCE login for CLI, the auth...
# ask-ai
b
@ancient-wolf-19325 During PKCE login for CLI, the auth on browser was succeeded, but the CLI showed an error: token parse error id token err: access token err: oidc: malformed jwt: go-jose/go-jose: compact JWS format must have three parts, access token err: [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata
j
The error "oidc: malformed jwt: go-jose/go-jose: compact JWS format must have three parts" during PKCE login indicates that the access token or ID token returned is not a valid JWT (it may be missing parts or is not in the expected format). This is often caused by a misconfiguration in the OAuth2/OpenID Connect provider or the Flyte client setup, such as using the wrong client type, incorrect scopes, or a misconfigured redirect URI. The error "[JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata" further suggests that the CLI could not extract or validate the required token from the authentication response. To resolve this, ensure that your IdP (e.g., Okta, Keycloak) client is configured to issue JWTs for access and ID tokens, and that the Flyte configuration (clientId, redirectUri, scopes) matches the IdP settings. Also, verify that the correct grant types and response types are enabled for the client, and that the PKCE flow is supported. For more details on troubleshooting and required settings, see the related discussions and configuration examples in this issue and this issue. Would you like a step-by-step breakdown of how to check and fix these settings? Sources: - https://github.com/flyteorg/flyte/issues/4592 - https://github.com/flyteorg/flyte/issues/4955 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 Yes
j
Here’s a step-by-step breakdown to diagnose and fix the PKCE login error: 1. Check IdP Client Configuration - Ensure your OAuth2/OpenID Connect client (in Okta, Keycloak, etc.) is set to issue JWTs for both access and ID tokens. - The client should be of the correct type: "public" for CLI tools like 
flytectl
(no client secret), and "confidential" for backend services (with client secret). 2. Verify Scopes and Grant Types - The scopes requested by Flyte (e.g., 
openid
, 
profile
, 
offline_access
, 
all
) must be allowed and mapped in your IdP. - For Keycloak, the scope for refresh tokens is 
offline_access
, not just 
offline
. - The client must allow the "authorization_code" grant type and support PKCE. 3. Check Redirect URI - The redirect URI in your IdP client configuration must exactly match the one used by Flyte CLI (e.g., 
http://localhost:53593/callback
). - Any mismatch will cause the token exchange to fail. 4. Inspect the Token Format - After authentication, decode the access and ID tokens (they should be three-part JWTs, separated by dots). - If the access token is not a JWT (sometimes IdPs return opaque tokens), configure your IdP to issue JWTs or use the ID token for authentication if supported. 5. Flyte Configuration - In your 
~/.flyte/config.yaml
, ensure the 
admin
section has the correct 
authType: Pkce
, 
clientId
, and 
redirectUri
. - Example:
Copy code
admin:
       endpoint: <your_flyteadmin_endpoint>
       authType: Pkce
       clientId: flytectl
       redirectUri: http://localhost:53593/callback
6. Logs and Error Messages - If you see errors like "Could not retrieve id token from metadata" or "malformed jwt," it usually means the token is not present or not a JWT. - Double-check the IdP logs and Flyte logs for more details. 7. Common Pitfalls - For Keycloak, ensure the client scope is set to 
offline_access
and not just 
offline
. - For Okta, make sure the API access management is enabled and the correct scopes are assigned. If you follow these steps and still encounter issues, please provide your IdP client configuration and the relevant Flyte config for further troubleshooting. Sources: - https://github.com/flyteorg/flyte/issues/4592 - https://github.com/flyteorg/flyte/issues/4955 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 For Flyte console login, why accessing /login?... it returns 403 directly? What config should I set to enable Flyte console (UI) PKCE?
j
A 403 error when accessing 
/login
in Flyte Console usually means authentication is enabled but not properly configured, or the console cannot reach the IdP or FlyteAdmin. To enable PKCE for Flyte Console, you must configure authentication in your Flyte deployment with the correct OIDC settings, clientId, and redirect URIs, and ensure your IdP client allows the PKCE flow and the redirect URI matches 
/callback
on your Flyte domain. Specifically, set 
auth.enabled: true
and 
auth.oidc.baseUrl
, 
clientId
, and 
clientSecret
in your Helm values file, and ensure 
redirectUri
is set to 
https://<your-flyte-domain>/callback
in your IdP. See the official authentication configuration guide and Keycloak example for details. Would you like a step-by-step example for a specific IdP (e.g., Keycloak, Okta, Google)? Sources: - Keycloak example - official authentication configuration guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 Can you give an example for the Flyte Console (not CLI) PKCE auth setup?
j
To enable PKCE authentication for Flyte Console (UI), configure your Flyte deployment with authentication enabled and set up OIDC parameters for your IdP. In your Helm values (e.g., 
values.yaml
), set:
Copy code
auth:
  enabled: true
  oidc:
    baseUrl: https://<your-idp-domain>/auth/realms/<realm>
    clientId: <flyte-console-client-id>
    clientSecret: <client-secret-if-required>
  authorizedUris:
    - https://<your-flyte-domain>
  internal:
    clientId: flytepropeller
    clientSecret: <propeller-client-secret>
In your IdP (e.g., Keycloak), register a client for the Flyte Console with: - Redirect URI: 
https://<your-flyte-domain>/callback
- Grant type: Authorization Code (with PKCE enabled) - Scopes: 
openid
, 
profile
, 
offline_access
This setup ensures the Flyte Console uses PKCE for authentication. For more details, see the official authentication configuration guide. Sources: - official authentication configuration guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 where should the auth config be in? Shouldn't be configmap -> flyteadmin -> auth -> userAuth?
j
Yes, for Flyte Console PKCE authentication, the relevant configuration should be under 
configmap.adminServer.auth.userAuth
in your Helm values or Kubernetes config. This section defines the OIDC settings for user authentication, including the 
openId
baseUrl, clientId, scopes, and redirect URL. For example:
Copy code
configmap:
  adminServer:
    auth:
      userAuth:
        openId:
          baseUrl: https://<your-idp-domain>/auth/realms/<realm>
          clientId: <flyte-console-client-id>
          scopes:
            - openid
            - profile
            - offline_access
This matches the structure described in the official authentication configuration guide. Sources: - official authentication configuration guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 how to clear the login token cache?
j
It seems likely that for Flyte CLI tools (like flytectl or pyflyte), the login token cache is stored in the OS keyring. To clear the cache, you can remove the relevant credentials from your system keyring or use a tool like 
keyrings.alt
if you are on a headless machine. There is no explicit Flyte command to clear the token cache; you must manually delete the stored credentials as described in this discussion. Would you like detailed steps for a specific operating system or tool? Sources: - https://github.com/flyteorg/flyte/issues/4784 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 Yes. MacOS
j
It seems likely that on MacOS, Flyte CLI tools (like flytectl or pyflyte) store tokens in the system Keychain. To clear the login token cache, open the "Keychain Access" app, search for entries related to "flyte" or your Flyte endpoint, and delete those items. This will force the CLI to prompt for authentication again on the next use, as described in this discussion. Would you like a step-by-step walkthrough of the Keychain Access process? Sources: - https://github.com/flyteorg/flyte/issues/4784 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
Why I use pyflyte to login and I was redirected to the login page for login two times? @ancient-wolf-19325
j
When using pyflyte with certain setups (e.g., GCP Identity-Aware Proxy or similar proxies), you may be prompted to log in twice: once for the proxy (e.g., IAP) and once for Flyte itself. This happens because pyflyte sends both a proxy-authorization header (for the proxy) and an authorization header (for Flyte), and each system requires its own authentication flow. This "double login" is a known behavior and is discussed in this issue and the flytekit IAP plugin README. Usually, after the initial double login, refresh tokens prevent repeated prompts unless one of the tokens expires. Would you like more detail on how to reduce or avoid this behavior? Sources: - https://github.com/flyteorg/flyte/issues/3965 - https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-identity-aware-proxy/README.md 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 How to enable user login but disable the auth between Flyte components?
j
It is not currently possible to enable user login (authentication) for Flyte Console or CLI while disabling authentication between Flyte internal components (such as FlyteAdmin and FlytePropeller). The authentication setting is global: when enabled, it applies to both user-facing and internal component communication. This is confirmed in this issue discussion, where it is stated that "auth is all or nothing right now." Would you like more detail on possible workarounds or future roadmap? Sources: - https://github.com/flyteorg/flyte/issues/1977 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 What's the auth configuration will Flyte components use to communicate with each other? Where to config the client id and client secret?
j
Flyte components (like FlytePropeller and FlyteAdmin) use OAuth2 client credentials flow to authenticate with each other. The client ID and client secret for internal communication are configured under the 
internal
section of the 
auth
config in your Helm values or config file. For example:
Copy code
auth:
  enabled: true
  internal:
    clientId: flytepropeller
    clientSecret: <your-client-secret>
    clientSecretHash: <bcrypt-hash-of-secret>
This configuration is described in the official authentication configuration guide. Would you like a step-by-step example for a specific deployment method? Sources: - official authentication configuration guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 How do I config if I do not user internal idp, but external
j
If you use an external IdP (not Flyte's internal auth server), set 
auth.appAuth.authServerType: External
and configure 
externalAuthServer
with your IdP's base URL, metadata URL, and allowed audience. Also, set the 
internal
section with the clientId and clientSecret provided by your IdP for FlytePropeller (or other components). Example:
Copy code
auth:
  enabled: true
  appAuth:
    authServerType: External
    externalAuthServer:
      baseUrl: https://<your-idp-domain>/auth/realms/<realm>
      metadataUrl: .well-known/openid-configuration
      allowedAudience: https://<your-flyte-deployment-URL>
  internal:
    clientId: <client-id-from-idp>
    clientSecret: <client-secret-from-idp>
    clientSecretHash: <bcrypt-hash-if-required>
See the official guide for more details. Would you like a full example for a specific IdP (e.g., Keycloak, Okta, Azure)? Sources: - official guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 Do I need to, and how do I get and config clientSecretHash if I use external auth server type?
j
If you use an external auth server, you do not need to set `clientSecretHash`; instead, you set the 
clientSecret
(the plain secret from your IdP) in the 
internal
section. The 
clientSecretHash
is only required if you use Flyte's internal auth server. For external IdPs, provide the 
clientId
and 
clientSecret
as given by your IdP, as shown in the official guide. Would you like a step-by-step example for a specific IdP? Sources: - official guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 But I use Flyte-core, do I still need to config clientSecretHash if I use external auth server type?
j
No, if you use Flyte-core with an external auth server, you do not need to set `clientSecretHash`; you only need to set the 
clientId
and 
clientSecret
(the plain secret from your IdP) in the 
secrets.adminOauthClientCredentials
section. The 
clientSecretHash
is only required for the internal auth server. This is confirmed in the Flyte-core values.yaml and related discussions. Would you like a sample configuration for Flyte-core with an external IdP? Sources: - Flyte-core values.yaml 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 how to config different client id and secret for each propeller?
j
To configure different client IDs and secrets for each FlytePropeller instance (e.g., in a multi-cluster setup), you must set unique 
clientId
and 
clientSecret
values for each propeller in their respective cluster's configuration. Each propeller's Helm values or Kubernetes secret should contain its own credentials, as described in the official guide. This allows each propeller to authenticate independently with the external IdP. For each cluster, update the 
secrets.adminOauthClientCredentials
and 
configmap.adminServer.auth.internal
sections with the specific 
clientId
and 
clientSecret
for that propeller. Would you like a concrete example for multi-cluster configuration? Sources: - official guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 yes
j
Here’s a concrete example for configuring different client IDs and secrets for each FlytePropeller in a multi-cluster setup: For each data plane (cluster), in its Helm values file (e.g., 
values-dataplane1.yaml
, 
values-dataplane2.yaml
), set:
Copy code
secrets:
  adminOauthClientCredentials:
    enabled: true
    clientId: <unique-client-id-for-this-propeller>
    clientSecret: <unique-client-secret-for-this-propeller>

configmap:
  adminServer:
    auth:
      internal:
        clientId: <unique-client-id-for-this-propeller>
        clientSecret: <unique-client-secret-for-this-propeller>
Each FlytePropeller instance will use its own client credentials to authenticate with FlyteAdmin and the external IdP. This approach is described in the official guide. Let me know if you need a full multi-cluster example or more details on secret management. Sources: - official guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 yes
j
I'm here to help with any questions about Flyte! Could you please provide more details or clarify your question? If you have a specific question about Flyte configuration, deployment, or usage, let me know and I'll do my best to assist you. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
b
@ancient-wolf-19325 I need a full multi-cluster example or more details on secret management.
j
For a full Flyte multi-cluster setup, each data plane (Kubernetes cluster running FlytePropeller) should have its own unique client ID and secret for authentication with the control plane (FlyteAdmin) and your external IdP. You must create a Kubernetes Secret in each data plane cluster with the credentials, and reference it in the Helm values for that cluster. The control plane must also be configured to recognize and authenticate each data plane’s credentials. Example for two data planes: 1. In each data plane cluster, create a secret:
Copy code
kubectl create secret generic flyte-secret-auth \
  --from-literal=client_secret=<UNIQUE_CLIENT_SECRET> \
  -n flyte
2. In each data plane’s Helm values (e.g., 
values-dataplane1.yaml
):
Copy code
secrets:
  adminOauthClientCredentials:
    enabled: true
    clientId: <UNIQUE_CLIENT_ID>
    clientSecret: <UNIQUE_CLIENT_SECRET>
configmap:
  adminServer:
    auth:
      internal:
        clientId: <UNIQUE_CLIENT_ID>
        clientSecret: <UNIQUE_CLIENT_SECRET>
3. In the control plane’s Helm values (e.g., 
values-controlplane.yaml
), configure clusters and authentication:
Copy code
configmap:
  clusters:
    labelClusterMap:
      dataplane1:
        - id: dataplane1
          weight: 1
      dataplane2:
        - id: dataplane2
          weight: 1
    clusterConfigs:
      - name: "dataplane1"
        endpoint: https://<dataplane1-api>
        enabled: true
        auth:
          type: "file_path"
          tokenPath: "/var/run/credentials/dataplane1_token"
          certPath: "/var/run/credentials/dataplane1_cacert"
        inCluster: false
      - name: "dataplane2"
        endpoint: https://<dataplane2-api>
        enabled: true
        auth:
          type: "file_path"
          tokenPath: "/var/run/credentials/dataplane2_token"
          certPath: "/var/run/credentials/dataplane2_cacert"
        inCluster: false
  adminServer:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: https://<your-idp-domain>/auth/realms/<realm>
          metadataUrl: .well-known/openid-configuration
          allowedAudience: https://<your-flyte-deployment-URL>
4. Mount the cluster credentials in the control plane: Create a secret in the control plane cluster with the bearer tokens and CA certs for each data plane, and mount it as 
/var/run/credentials
in the FlyteAdmin pod. 5. Each FlytePropeller instance will use its own clientId/clientSecret to authenticate with FlyteAdmin and the IdP. For more details, see the official multi-cluster deployment guide and authentication configuration. Would you like a step-by-step walkthrough for a specific cloud provider or more on secret manager plugins? Sources: - official multi-cluster deployment guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
3 Views
Open in Slack
PreviousNext
Powered by