https://flyte.org logo
Join Slack
Powered by
Hi Community, I tried to integrate self-hosted Git...
# flyte-deployment
s
Hi Community, I tried to integrate self-hosted Gitlab as an external OAuth2 Authorization Server but stuck at 
CSRF_TOKEN_VALIDATION_FAILED
, any clues on how to debug the problem? The flyte is deployed on self-hosted k8s cluster with an internal domain name and HTTP access (not HTTPS) using template generated by 
flyte-core
helm chart (
helm template flyte-core -f values.yaml
). The OIDC seems to work (After a fresh deployment, when opening flyte console, it will redirect to the gitlab authorization page, and after approval it can redirect back to flyte console page with username show on the top-right corner), but when I try to register workflow according to the tutorial [1] with command 
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
, the gitlab authorization page is prompted successfully but when redirect back to 
<http://flyte.example.com/callback|flyte.example.com/callback>
page, it complains about 
401
error. And the log of 
flyteadmin
shows something like
Copy code
{
  "json": {},
  "level": "error",
  "msg": "Invalid CSRF token cookie [CSRF_TOKEN_VALIDATION_FAILED] CSRF token does not match state 2r4rcd3npg, 3237e1083ec0ae2bd20acbe8a5817d18475faaee5a060d2184ab7ffddd151290 vs OXpoczQyanRxcW43c3hnZ3RjbnBnZjZrNnptMnA2dDY",
  "ts": "2023-01-29T06:43:34Z"
}
There are several small questions that might be relevant: 1. In the auth doc [2], there is a comment for 
redirectUri: <http://localhost:53593/callback>
saying that 
This should not change
, but I suppose it is for sandbox deployment so I changed it to the domain name something like 
<http://flyte.example.com/callback>
. Otherwise, it will complains about invalid redirect uri when authorize on gitlab page. 2. For the 
scopes
, according to the gitlab doc [3], I only set it to 
read_user
. 3. Any approach to show more logs for 
flytectl
CLI? I tried to set the log level to 
0
, but it does not show anything. 4. Any approach to show more logs for 
flyteadmin
service? I found there is a 
flyteadmin.extraArgs
in the template but does not know how to inject 
--logger.level 0
. The relevant 
values.yaml
for auth is shown as following (with some sensitive info masked):
Copy code
configmap:
  adminServer:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          metadataUrl: .well-known/openid-configuration
        thirdPartyConfig:
          flyteClient:
            clientId: <client_id generated by gitlab>
            redirectUri: <http://flyte.example.com/callback>
            scopes:
              - read_user
      authorizedUris:
        - <http://flyte.example.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://git.example.com>
          clientId: <client_id generated by gitlab>
          scopes:
            - openid
    server:
      security:
        useAuth: true
flyteadmin:
  secrets:
    oidc_client_secret: <client_secret generated by gitlab>
secrets:
  adminOauthClientCredentials:
    clientId: <client_id generated by gitlab>
    clientSecret: <client_secret generated by gitlab>
[1] https://docs.flyte.org/projects/cookbook/en/latest/auto/larger_apps/larger_apps_deploy.html#build-deploy-your-application-to-the-cluster [2] https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html [3] https://docs.gitlab.com/ee/integration/oauth_provider.html#view-all-authorized-applications
f
The number of auth integrations is crazy. But I know someone else is using gitlab
s
Will it be possible have some configuration examples or clues to debug the problem?
f
Let me add a few auth experts here
Cc @icy-agent-73298 / @high-park-82026
s
Another information is that we are using HAProxy instead of Nginx for Ingress, that might be the reason for the csrf token error, since I am not an expert, I have already involved one of my colleagues for help. Just want to confirm the config looks good and if there is any recommendation to debug the problem.
Sorry, I missed one import aspect, the client configuration. The 
config.yaml
I use now is
Copy code
admin:
  authType: Pkce
  endpoint: dns:///flyte.example.com
  insecure: false
  insecureSkipVerify: true
logger:
  level: 0
  show-source: true
With this config, the gitlab authorization page can be prompted successfully but 
401
error after callback to 
flyteadmin
. I also tried the 
ClientSecret
type but failed. It complains about 
JWT_VERIFICATION_FAILED
or 
500
error for gitlab. The gitlab authorization page will not shown.
i
Hi @strong-barista-29632 , please check my comments on these
In the auth doc [2], there is a comment for 
redirectUri: <http://localhost:53593/callback>
saying that 
This should not change
, but I suppose it is for sandbox deployment so I changed it to the domain name something like 
<http://flyte.example.com/callback>
.
This needs to remain the same 
<http://localhost:53593/callback>
as this is the url where flytectl recives a callback to exchange auth code for the token in PKCE flow started by it. So this would remain same irrespective of sandbox.
Otherwise, it will complains about invalid redirect uri when authorize on gitlab page.
You don’t have to configure this callback url in gitlab . Can you more details here
For the 
scopes
, according to the gitlab doc [3], I only set it to 
read_user
.
Currently its required according to our current set of supported providers to define scope name 
all
. other providers like okta support adding new scopes for oauth apps
Any approach to show more logs for 
flytectl
CLI? I tried to set the log level to 
0
, but it does not show anything.
You can pass arguments --logger.level 6
Any approach to show more logs for 
flyteadmin
service? I found there is a 
flyteadmin.extraArgs
in the template but does not know how to inject 
--logger.level 0
.
you can edit you flyte admin base configmap and add logger configuration with level 6
Copy code
logger.yaml: |
      logger:
        level: 6
🙏 1
❤️ 1
s
Hi @icy-agent-73298, thanks for the detailed reply! I will try it soon and come back then.
👍 1
Hi @icy-agent-73298, I am still a little confused about the 
redirectUri
part. Let us assume the flyte is deployed with an internal domain name, for example, 
<http://flyte.example.com>
. The 
redirectUri
need to be configured for both 
flyteadmin
and gitlab Application [1]. And during the OAuth process [2], web browser will firstly redirect to the Gitlab authorization page, And after clicking 
Authorize
, it will redirect back to the configured 
redirectUri
. If we use 
<http://localhost:53593/callback>
then it will be an invalid address since the service is deploy with 
<http://flyte.example.com>
instead. Any comments? [1] https://docs.gitlab.com/ee/integration/oauth_provider.html#create-a-user-owned-application [2] https://docs.gitlab.com/ee/api/oauth2.html#authorization-code-with-proof-key-for-code-exchange-pkce
t
Are you seeing any error when the 
redirectUri
is 
<http://localhost:53593/callback>
?
s
Yes, firstly, in the configuration of Gitlab Application, I have to set the Redirect URI to the same one, a.k.a, 
<http://localhost:53593/callback>
otherwise it will complains about invalid redirect uri. And then after I click the 
Authorize
button when being prompted to Gitlab Authorization page, it will redirect back to 
<http://localhost:53593/callback>
in web browser which is a invalid address since the flyte is deployed on the server.
i
Hi @strong-barista-29632, Sorry for the confusion. Seems the docs would need to be improved here There are three aoauth gitlab apps you need to configure to use with flyte Repasting the configmap you posted along with places where these clientId are defined and there purpose.
Copy code
configmap:
  adminServer:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          metadataUrl: .well-known/openid-configuration
        thirdPartyConfig:
          flyteClient:
            clientId: <client_id generated by gitlab>              => This clientId is used for flytectl client and should be configured as native ouath app 
            redirectUri: <http://flyte.example.com/callback>         => the redirectURI here should be <http://localhost:53593/callback> since flytectl starts a http server to receive the callback in this flow
            scopes:
              - read_user
      authorizedUris:
        - <http://flyte.example.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://git.example.com>
          clientId: <client_id generated by gitlab>                => this clientId is used for flyteconsole or UI client which uses client credentials flow and it secret is stored in oidc_client_secret . The redirect URI for this app should be <http://flyte.example.com/callback> or wherever your <flyte domain is configured>/callback
          scopes: 
            - openid
    server:
      security:
        useAuth: true
flyteadmin:
  secrets:
    oidc_client_secret: <client_secret generated by gitlab> => used as secret for openId.clientId
secrets:
  adminOauthClientCredentials:
    clientId: <client_id generated by gitlab>           => this clientId is used by flytepropeller and scheduler . this should also use the same callback used by flyteconsole   <http://flyte.example.com/callback> or wherever your <flyte domain is configured>/callback
    clientSecret: <client_secret generated by gitlab>   => secret for the above clientId
the redirectURI http://localhost:53593/callback is only to be configured for flytectl app and when you run any command from flytectl it will open a browser to do authorization using gitlab, while the browser is running , flytectl will be running a http callback server on port 53593 and when you complete the authorization flow in the browser it will redirect to this http callback endpoint and exchange the auth token with the gitlab .
We can jump on a call if its easier and we can coordinate on a time
s
Thanks @icy-agent-73298 for the detailed explanation, will come back after some further investigation.
Hi @icy-agent-73298, good news is that I can see authentication result page when using flytectl with 
Pkce
auth type, but the auth process still does not work. The config is something like
Copy code
admin:
  authType: Pkce
  endpoint: dns:///flyte.example.com
  insecure: false
  insecureSkipVerify: true
And the error log is
Copy code
{"json":{"src":"token_source_provider.go:140"},"level":"error","msg":"Error fetching token using auth flow due to error while exchanging auth code due to oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method.\"}","ts":"2023-01-31T18:07:09+08:00"}
Error: authentication error! Original Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken, Auth Error: error while exchanging auth code due to oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method."}
{"json":{"src":"main.go:13"},"level":"error","msg":"authentication error! Original Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken, Auth Error: error while exchanging auth code due to oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method.\"}","ts":"2023-01-31T18:07:09+08:00"}
BTW, I failed to find any place to set the 
client_secret
for 
flytectl
in helm chart template, that may be the cause for the unauthenticated problem. I also tried 
ClientSecret
auth type, but it will get 500 internal error from gitlab. The config is something like
Copy code
admin:
  authType: ClientSecret
  clientId: <client id for flytectl>
  clientSecretLocation: <path of client secret for flytectl>
  endpoint: dns:///flyte.example.com
  insecure: false
  insecureSkipVerify: true
  scopes: [ "openid", "read_user"]
The relevant error log is
Copy code
{
  "json": {
    "src": "main.go:13"
  },
  "level": "error",
  "msg": "Connection Info: [Endpoint: dns:///flyte.example.com, InsecureConnection?: false, AuthMode: ClientSecret]: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 500 Internal Server Error\nResponse: <!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n  <title>Something went wrong (500)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: 400;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: 400;\n      line-height: 28px;\n    }\n\n    hr {\n      max-width: 800px;\n      margin: 18px auto;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n\n    img {\n      max-width: 40vw;\n      display: block;\n      margin: 40px auto;\n    }\n\n    a {\n      line-height: 100px;\n      font-weight: 400;\n      color: #4A8BEE;\n      font-size: 18px;\n      text-decoration: none;\n    }\n\n    .container {\n      margin: auto 20px;\n    }\n\n    .go-back {\n      display: none;\n    }\n\n  </style>\n</head>\n\n<body>\n  <a href=\"/\">\n    <img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=\"\n       alt=\"GitLab Logo\" />\n  </a>\n  <h1>\n    500\n  </h1>\n  <div class=\"container\">\n    <h3>Whoops, something went wrong on our end.</h3>\n    <hr />\n    <p>Try refreshing the page, or going back and attempting the action again.</p>\n    <p>Please contact your GitLab administrator if this problem persists.</p>\n    <a href=\"javascript:history.back()\" class=\"js-go-back go-back\">Go back</a>\n  </div>\n  <script>\n    (function () {\n      var goBack = document.querySelector('.js-go-back');\n\n      if (history.length > 1) {\n        goBack.style.display = 'inline';\n      }\n    })();\n  </script>\n</body>\n</html>\n",
  "ts": "2023-01-31T16:05:02+08:00"
}
Any comments are welcome. And Thanks in advance!
i
Hi @strong-barista-29632, Seems the gitlab oauth app is misconfigured according to the errors you pasted • PKCE based apps don’t have clientSecret. Seems the app has been incorrectly been created in gitlab . I couldn’t find the documentation from gitlab docs to create such apps but this is corresponding reference from OKTA provider • The error 
Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method
indicates this misconfiguration. Seems it attempting to do a PKCE flow on an gitlab oauth app which has been created as client secret app • When you are using ClientSecret of flytectl and getting 500 error which is again not an expected flow and hence it might be throwing this from gitlab. I would suggest getting right docs to setup gitlab oauth PKCE app (no client secret) Other option is to use client secret based app in which you can try reusing the clientId and secret which is used for flytepropeller and scheduler i.e this one
Copy code
adminOauthClientCredentials:
    clientId: <client_id generated by gitlab>
    clientSecret: <client_secret generated by gitlab>
The second option is suited for CI usescases where you just have CI server authenticating with gitlab whereas if you are deploying flytectl to multiple users machines with browser access, PKCE based oauth is the preferred approach
s
Hi @icy-agent-73298, thanks for the explanation. I have to admit I am not an expert for OAuth. I will try to investigate more. Thanks again for all the patient help.
i
Np. Unfortunately gitlab as auth provider hasn’t been tried in the flyte community yet but but let us know how it goes and will be there to help you on it
s
Hi @icy-agent-73298, thanks for the kindness. Just to update the latest progress. For 
Pkce
auth type, Good news is that I can see the successful authentication result page from 
localhost
with info 
Cool! Your authentication was successful and you can close the window.
. The key point for no client secret authentication is to uncheck the 
Confidential
option (refer to the attached screenshot). But the auth process still has some problem. The error log is something like below:
Copy code
{"json":{"src":"auth_flow_orchestrator.go:77"},"level":"info","msg":"Opening the browser at <https://git.example.com/oauth/authorize?client_id=><client_id for flytectl with localhost redirect uri>\u0026redirect_uri=http%3A%2F%2Flocalhost%3A53593%2Fcallback\u0026response_type=code\u0026scope=email+openid+profile+read_user\u0026code_challenge=gYYRv7I-Eo2E3xiEbF2iSaIJl5hVVOeXJBzI0NQ-6l4\u0026code_challenge_method=S256\u0026nonce=cG1sd2NobG5zNzYycjJrY3ZqeDRrN3ZsNWQ1NDdkang\u0026state=cTduc242bTZja3FuNWdtczJ2ZmNuMmI2czdoc3hkdHo","ts":"2023-02-02T11:17:00+08:00"}
Error: Connection Info: [Endpoint: dns:///flyte.example.com, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///flyte.example.com, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2023-02-02T11:17:10+08:00"}
I searched on the Google and find a StackOverflow answer [1] saying that 
email
scope need to be checked, but I tried it with even more scopes and it does not work. For the 
ClientSecret
auth type, I tried to reuse the 
client_id
and 
client_secret
for flytepropeller and scheduler, but it still get 500 error from gitlab. For reminder, the auth related 
values.yaml
for 
flyte-core
helm chart template I use now is
Copy code
configmap:
  adminServer:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          metadataUrl: .well-known/openid-configuration
        thirdPartyConfig:
          flyteClient:
            clientId: <client_id for flytectl and SAME with flyteproerller and scheduler for ClientSecret auth type>
            redirectUri: <http://localhost:53593/callback>
            scopes:
              - email
              - openid
              - profile
              - read_user
      authorizedUris:
        - <http://flyte.example.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://git.example.com>
          clientId: <client_id for flyteconsole> 
          scopes:
            - email
            - openid
            - profile
            - read_user
    server:
      security:
        useAuth: true
flyteadmin:
  secrets:
    oidc_client_secret: <client_secret for flyteconsole>
secrets:
  adminOauthClientCredentials:
    clientId: <client_id for flytepropeller and scheduler>
    clientSecret: <client_secret for flytepropeller and scheudler>
the client config for 
Pkce
and 
ClientSecret
is almost the same as before
Copy code
admin:
  authType: Pkce
  endpoint: dns:///flyte.example.com
  insecure: false
  insecureSkipVerify: true
and
Copy code
admin:
  authType: ClientSecret
  clientId: <SAME client_id for flytepropeller and scheduler>
  clientSecretLocation: /etc/secrets/client_secret
  endpoint: dns:///flyte.example.com
  insecure: false
  insecureSkipVerify: true
[1] https://stackoverflow.com/a/67213376
Also for the gitlab applications, I created three right now: 1. flytectl (only used for Pkce client authentication) a. Redirect URI: http://localhost:53593/callback b. Confidiential: Unchecked c. Scopes: read_user, openid, profile, email 2. flyteconsole a. Redirect URI: http://flyte.example.com/callback b. Confidential: Checked c. Scopes: read_user, openid, profile, email 3. flytepropeller_scheduler a. Redirect URI: (tried both localhost and flyte.example.com here) b. Condifential: Checked c. Scopes: read_user, openid, profile, email
BTW, 
ClientSecret
auth type should be a must have option since we have scenarios for CICD, e.g., Daily Test, auto finetune with incremental data. Also algorithm engineers tend to work on server over ssh with no easy (though it could be done via ssh forward) web browser access.
f
@strong-barista-29632 in short are you still stuck
s
Hi @freezing-airport-6809, thanks for the caring. @icy-agent-73298 is helping me on that. Currently, seems there needs a new release of flytectl with the latest flyteidl library. We will continue on this tomorrow. It is not a hurry blocking issue, so we will just keep tracking on it.
i
From flyteadmin logs which was shared , we saw that authserver is sending opaque token and hence parsing of the token is failing since its not in JWT format and this causes flyteadmin to reject the token
Copy code
{"json":{"src":"handlers.go:248"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: oidc: malformed jwt: square/go-jose: compact JWS format must have three parts","ts":"2023-02-02T06:34:38Z"}
{"json":{"src":"token.go:103"},"level":"deb
This has happened in the past when the authserver is not sent the right audience config and which has been fixed to allow setting this from client config in flytectl (We dont face this internally since we have okta auth server with default audience set to hosted flyte endpoint. Certain auth providers like gitlab and auth0 dont have an option to set this default audience and expect the client to send in this when requesting the auth token) We will upgrade flytectl with latest flyteidl and test this out with passing audience config and take it from there.
🙏 1
f
I saw you merged, do we think that’s it
i
@strong-barista-29632 We can do 
flytectl upgrade
and use the config with audience . Also this change to the audience would need to be done to all flyteadmin clients , ie. scheduler, propeller and later flytekit too (PR in progress for flytekit https://github.com/flyteorg/flytekit/pull/1403) We can first verify that this audience issue and then update the values file for propeller and admin to use the same .
f
Cc @strong-barista-29632
s
Thanks @freezing-airport-6809 and @icy-agent-73298, I will try it later. Suppose to reply early next week.
👍 1
Hi @icy-agent-73298, I tried the new version 
0.6.27
of flytectl and it seems to be the same error. Config for flytectl
Copy code
admin:
  authType: Pkce
  endpoint: dns:///flyte.example.com
  insecure: false
  insecureSkipVerify: true
  audience: <http://flyte.example.com>
logger:
  level: 6
  show-source: true
Error log for flytectl
Copy code
{"json":{"src":"auth_flow_orchestrator.go:77"},"level":"info","msg":"Opening the browser at <https://git.example.com/oauth/authorize?client_id=88151b597efcd5adbfe86fcf54c2a3eb685daa552a41f4bfe4888b95bb10c571>\u0026redirect_uri=http%3A%2F%2Flocalhost%3A53593%2Fcallback\u0026response_type=code\u0026scope=email+openid+profile+read_user\u0026code_challenge=c6ja-6ytu4dOTvNfvwgwjnjOmw10niejG7To47Jqymw\u0026code_challenge_method=S256\u0026nonce=bDkyeGo2c25jdmZucDYyNDhnbDJzbGhuanI1aGd6MmI\u0026state=aHo1ZHh4ZHp2NGZxaGhkdGJycHZzZ240bHhoZDh2dHQ","ts":"2023-02-06T13:17:41+08:00"}
{"json":{"src":"version.go:103"},"level":"debug","msg":"Failed to get version of control plane rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken: \n","ts":"2023-02-06T13:17:46+08:00"}
{"json":{"src":"version.go:81"},"level":"debug","msg":"rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2023-02-06T13:17:46+08:00"}
ConfigMap for flyteadmin, the 
values.yaml
Copy code
configmap:
  adminServer:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          allowedAudience: <http://flyte.example.com>
          metadataUrl: .well-known/openid-configuration
        thirdPartyConfig:
          flyteClient:
            audience: <http://flyte.example.com>
            clientId: 88151b597efcd5adbfe86fcf54c2a3eb685daa552a41f4bfe4888b95bb10c571
            redirectUri: <http://localhost:53593/callback>
            scopes:
              - email
              - openid
              - profile
              - read_user
      authorizedUris:
        - <http://flyte.example.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://git.example.com>
          clientId: 3483bdaa483ebe93638fe637481013c5ca869dd67dceaf45ccf6351ed61ba0ca
          scopes:
            - email
            - openid
            - profile
            - read_user
    server:
      security:
        useAuth: true
Error log for flyteadmin
Copy code
{"json":{"src":"handlers.go:238"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"token.go:83"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"handlers.go:248"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"token.go:103"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor []method [/flyteidl.service.AuthMetadataService/GetPublicClientConfig]\n","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor []method [/flyteidl.service.AuthMetadataService/GetOAuth2Metadata]\n","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor []method [/flyteidl.service.AuthMetadataService/GetPublicClientConfig]\n","ts":"2023-02-06T05:17:41Z"}
{"json":{"src":"handlers.go:210"},"level":"debug","msg":"Found existing metadata Bearer e54a72ad7f9e782bdcd0febfd146dc83f5b2340a789892ada5808c58df58eadb","ts":"2023-02-06T05:17:46Z"}
{"json":{"src":"handlers.go:238"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2023-02-06T05:17:46Z"}
{"json":{"src":"handlers.go:248"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: oidc: malformed jwt: square/go-jose: compact JWS format must have three parts","ts":"2023-02-06T05:17:46Z"}
{"json":{"src":"token.go:103"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2023-02-06T05:17:46Z"}
Besides, I can confirm the token is valid. The following command can get valid response:
Copy code
$ curl --header "Authorization: Bearer e54a72ad7f9e782bdcd0febfd146dc83f5b2340a789892ada5808c58df58eadb" "<https://git.example.com/oauth/token/info>"
i
Hi @strong-barista-29632, seems from the above curl command you shared that the token return by your configured gitlab identity provider is not issuing JWT tokens . Identity tokens are used by flyteadmin to know the users identity info once he is authenticated and verify the Identity tokens signature. Since in your case the the auth provider is not issuing JWT and hence it failing in flyteadmin to parse this token https://github.com/go-jose/go-jose/blob/b5b8c96bd255ff55fb1a6f1008e92dd6e7e92f05/jws.go#L277 This is the function which calls to verify the format for token https://github.com/go-jose/go-jose/blob/b5b8c96bd255ff55fb1a6f1008e92dd6e7e92f05/jws.go#L79 Can you check your gitlab setup to see what it takes for it to issue JWT tokens . Also according to RFC these tokens need to be JWT tokens https://openid.net/specs/openid-connect-core-1_0.html#IDToken
s
Hi @icy-agent-73298, thanks for the reply, and this is the OAuth2 API for Gitlab [1], seems there is no option for jwt token, can we conclude that, at least currently, gitlab is not supported? Again, I am not an expert of OAuth, just want to have a double confirm. [1] https://docs.gitlab.com/ee/api/oauth2.html
According to the Overview of Authentication in Flyte [1], oidc is used for user’s authentication while OAuth2 is used for client, so I suppose we might support access token besides jwt? Also according to the log, there is an error log 
Could not retrieve bearer token from metadata
, will this be the actual cause? Which should be related to this code snippet [2], but since I am not familiar with Go, I failed to find anything unusual. [1] https://docs.flyte.org/en/v1.0.0/deployment/cluster_config/auth_setup.html#overview [2] https://github.com/flyteorg/flyteadmin/blob/master/auth/token.go#L81
225 Views
Open in Slack
PreviousNext
Powered by