Haytham Abuelfutuh

    Haytham Abuelfutuh

    3 weeks ago
    @Hampus Rosvall @Prafulla Mahindrakar Let’s discuss auth here
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    I just checked the configmaps for propeller and scheduler, there is a portion dedicated to client credentials auth flow, but I can’t seem to find an option to override the scopes here
    ~ k get configmap flyte-{propeller, scheduler}-config  
      admin.yaml: |
        admin:
          clientId: clientId
          clientSecretLocation: /etc/secrets/client_secret
          endpoint: flyteadmin:81
          insecure: true
    p

    Prafulla Mahindrakar

    3 weeks ago
    You can add it to the same section with key scopes which takes a list in yaml
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Right, so something like?
    admin:
          endpoint: flyteadmin:81
          insecure: true
          clientId: "{{ .Values.secrets.adminOauthClientCredentials.clientId }}"
          clientSecretLocation: /etc/secrets/client_secret
          scopes: ["myScopes"]
    p

    Prafulla Mahindrakar

    3 weeks ago
    yes. Or i guess yaml also uses this
    admin:
          endpoint: flyteadmin:81
          insecure: true
          clientId: "{{ .Values.secrets.adminOauthClientCredentials.clientId }}"
          clientSecretLocation: /etc/secrets/client_secret
          scopes: 
             - myscopes
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Cool, let’s try that
    p

    Prafulla Mahindrakar

    3 weeks ago
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Okay some progress, I get sent to Azure IdP and get redirected with
    Cool! Your authentication was successful and you can close the window.
    But I am getting this response now from the admin
    Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
    And in the logs from the admin it finds an existing metadata Bearer, but when I validate the token on jwt.io it says that the signature is invalid. Maybe I have not configured the PKCE app successfully?
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    3 weeks ago
    Hey @Hampus Rosvall do you have time to debug this live now?
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Ah, not by my computer atm.
    But I can try to dig into this more during the day tomorrow and we can have a call tomorrow if it’s not resolved?
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    3 weeks ago
    👍
    p

    Prafulla Mahindrakar

    3 weeks ago
    Hi @Hampus Rosvall, can you paste the logs from admin . And also what are the config.yaml for flytectl and your admin auth config. If it easier we can get on a call to debug this
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Hey, sure. By the way, is there a way to override the allowed audience for the PKCE auth flow?
    appAuth:
            authServerType: External
            externalAuthServer:
              baseUrl: <https://login.microsoftonline.com/tenant/v2.0>
              metadataUrl: .well-known/openid-configuration
              allowedAudience:
                - clientid
    This is what I am doing for the client credentials flow, and I am getting invalid audience error for the PKCE flow now
    I think the error is within my app for PKCE flow, and the scopes I am sending - so I am just trying to figure out how it should be configured.
    p

    Prafulla Mahindrakar

    3 weeks ago
    In your Azure authorization server can you check what is the audience configuration used. The allowedAudience field needs to match one set in the authorization server
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Yes, when I validate the token, the audience should equal the clientId.
    So I need to find a way to provide that during token validation, as the allowed audience is different for client credentials flow and PKCE due to two different apps
    https://github.com/flyteorg/flyteadmin/blob/f72aa348e4a548d38f5afc5dcbc49f56fe67a75f/auth/authzserver/provider.go#L138-L143 seems like the code checks if the audience from the claim is in the list of allowed audiences, so I might be able to add it
    p

    Prafulla Mahindrakar

    3 weeks ago
    The allowed audience can be a list and its usually the endpoint which flyte is hosted on And this what should be configured in the allowed audience in admin so that token sent by auth server has the same audience configured in admin
    yes exactly .
    For eg in okta this config is available directly on the auth server config page . This not specific to an app but the capability of auth server when it rolls out token and who its intended audience is .
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    flyteadmin-76f6476884-9vnxq flyteadmin panic: interface conversion: interface {} is string, not []interface {}
    flyteadmin-76f6476884-9vnxq flyteadmin
    allowedAudience:
                - propellerClientId
                - flytectlClientId
    p

    Prafulla Mahindrakar

    3 weeks ago
    Can you double quote those values . or use this representation
    key1: [value1,value2,value3,value4,value5]
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    I can’t right now, have a meeting in 15 min
    p

    Prafulla Mahindrakar

    3 weeks ago
    Ok sure. Let me know your availability .
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    I have set up an app according to these docs now. I am getting the following error
    Tokens issued for the 'Single-Page Application' client-type may only be redeemed via cross-origin requests
    which seems to be discussed in this thread e.g., this solution https://stackoverflow.com/a/67280275. Is this something I can add to the request?
    p

    Prafulla Mahindrakar

    3 weeks ago
    Can you check if the failure comes after this https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/pkce/handle_app_call_back.go#L44 And may be you would need to add the origin url when fetching the auth token at this place.
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Hm, how would I check that? The full error is from here
    {"json":{"src":"token_source_provider.go:121"},"level":"error","msg":"Error fetching token using auth flow due to error while exchanging auth code due to oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_request\",\"error_description\":\": Tokens issued for the 'Single-Page Application' client-type may only be redeemed via cross-origin requests
    p

    Prafulla Mahindrakar

    3 weeks ago
    So after opening the url it redirects to auth server to login and then on the callback hits the app call back hander with authCode to be exchanged for the token The app call back handler exchanges that here https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/pkce/handle_app_call_back.go#L44 So it seems in your case some policy blocks this exchange . Also looking at this i guess its forbidden to set this Origin header https://stackoverflow.com/questions/46808011/how-to-set-the-origin-request-header And recommended way is to allowCors. I remember we spoke about disabling this to test it out. Is it still disabled in your admin config. (allowCors) . With allowCors do you get this far in the flow or does it fail earlier .
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    This is the server portion of the
    server.yaml
    configMap:
    server:
          grpcPort: 8089
          httpPort: 8088
          security:
            allowCors: true
            allowedHeaders:
            - Content-Type
            allowedOrigins:
            - '*'
            secure: false
            useAuth: true
    So the error is still hit here:
    {"json":{"src":"token_source_provider.go:121"},"level":"error","msg":"Error fetching token using auth flow due to error while exchanging auth code due to oauth2: cannot fetch token:
    p

    Prafulla Mahindrakar

    3 weeks ago
    And what about the admin logs , can we check those too for the error.
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    flyteadmin-858647d86d-8f4h8 flyteadmin {"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-09-01T14:23:10Z"}
    flyteadmin-858647d86d-8f4h8 flyteadmin {"json":{"src":"token.go:83"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-09-01T14:23:10Z"}
    flyteadmin-858647d86d-8f4h8 flyteadmin {"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-09-01T14:23:10Z"}
    flyteadmin-858647d86d-8f4h8 flyteadmin {"json":{"src":"token.go:103"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-09-01T14:23:10Z"}
    p

    Prafulla Mahindrakar

    3 weeks ago
    The message is clipped can you paste the entire log . Is it the same issue
    Tokens issued for the 'Single-Page Application' client-type may only be redeemed via cross-origin requests
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    {"json":{"src":"token_source_provider.go:121"},"level":"error","msg":"Error fetching token using auth flow due to error while exchanging auth code due to oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_request\",\"error_description\":\"AADSTS9002327: Tokens issued for the 'Single-Page Application' client-type may only be redeemed via cross-origin requests'..}
    {"json":{"src":"client.go:188"},"level":"warning","msg":"Starting an unauthenticated client because: error while exchanging auth code due to oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_request\",\"error_description\":\"AADSTS9002327: Tokens issued for the 'Single-Page Application' client-type may only be redeemed via cross-origin requests"...}
    {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-01T16:23:09+02:00"}
    Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
    {"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-09-01T16:23:10+02:00"}
    The dots are just error codes any timestamps from Azure
    I am requesting the openid scope btw
    p

    Prafulla Mahindrakar

    3 weeks ago
    @Hampus Rosvall I think the app shouldn’t be an SPA but a native app . This is how we use it for Oktahttps://docs.microsoft.com/en-us/azure/active-directory/develop/v2-app-types
    Hampus Rosvall

    Hampus Rosvall

    3 weeks ago
    Ah awesome, will try that out. Thanks a lot
    When I use a mobile/native app I am actually getting to this line here i.e., the browser says I authenticated correctly. Here are the logs from the admin
    flyteadmin-858647d86d-mxzv5 flyteadmin {"json":{"src":"handlers.go:209"},"level":"debug","msg":"Found existing metadata Bearer JWT_TOKEN","ts":"2022-09-04T06:42:05Z"}
    flyteadmin-858647d86d-mxzv5 flyteadmin {"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-09-04T06:42:05Z"}
    flyteadmin-858647d86d-mxzv5 flyteadmin {"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: failed to verify id token signature","ts":"2022-09-04T06:42:05Z"}
    flyteadmin-858647d86d-mxzv5 flyteadmin {"json":{"src":"token.go:103"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-09-04T06:42:05Z"}
    Here is the problem most likely
    flyteadmin-858647d86d-mxzv5 flyteadmin {"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: failed to verify id token signature","ts":"2022-09-04T06:38:59Z"}
    After I am getting the successful auth message (i.e., this https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/pkce/handle_app_call_back.go#L52) in the browser I am seeing these logs in from flytectl
    Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
    {"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-09-04T08:42:05+02:00"}
    In the token header I am getting a
    nonce
    parameter, seems like OpenID Connect nonce and PKCE are not entirely the same auth flow as to where the token validation takes place
    p

    Prafulla Mahindrakar

    2 weeks ago
    Ok that one good news . So the auth flow completes and generated the token but the token is rejected by admin while verifying the signature Nonce is part of the header https://github.com/square/go-jose/blob/v2.5.1/shared.go#L181 So i am not sure if that causing an issue though. Unfortunately the same error is printed at two places , we will have to check bothhttps://github.com/coreos/go-oidc/blob/v2.2.1/jwks.go#L125 https://github.com/coreos/go-oidc/blob/v2.2.1/jwks.go#L140 Assuming first time nothing is in the cache so it should fail at the L140 Can you verify that if you open {baseUrl}/{metadataUrl} that you have property called jwks_uri / jwksUri Open the url pointed to jwks_uri / jwksUri. And note the kid property in the keys returned. There could be multiple keys Can you try pasting your saved token and use the access token from it and verify it on jwt.io . Does the signature show its verified over there . Also note the kid in the header. You can get kid also by running this code Also paste the token in here and can you check the KeyID printedhttps://go.dev/play/p/CR_fPsT36t_S Does it match the KID from jwks_uri data
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    Yeah the KID in the token matches the KID in the jwks_uri. However, the token says invalid signature when I try to validate it
    I think this has something to do with it
    p

    Prafulla Mahindrakar

    2 weeks ago
    Thanks for checking that. Lot of discussion and many ways folks have tried diff ways to resolve it. Audience and scope configuration can also lead to this looks like . Can you check what is the audience configured on the Azure ad auth server. In case of okta its the flyte-endpoint. Also the scope , may be introduce a new scope for the app detailed here https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/issues/609#issuecomment-887524901 @Haytham Abuelfutuh can you suggest what else should we try here.
    From the comments
    special processing
    from what i get means only the resource server (in our case flyteadmin) who is entitled to receive the token can understand the data .
    This err which is not printed anywhere may be can give us more clue https://github.com/coreos/go-oidc/blob/8d771559cf6e5111c9b9159810d0e4538e7cdc82/jwks.go#L135 You can try adding a log statement and consuming it in flyteadmin
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    I followed the steps from the github issue comment, and I am actually getting a valid token now, but the admin fails here after finding a bearer token, I remember us having this issue previously but I am not sure how we solved it
    flyteadmin-b6b9c465c-vgj57 flyteadmin {"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-09-06T06:15:34Z"}
    flyteadmin-b6b9c465c-vgj57 flyteadmin panic: interface conversion: interface {} is string, not []interface {}
    flyteadmin-b6b9c465c-vgj57 flyteadmin
    flyteadmin-b6b9c465c-vgj57 flyteadmin goroutine 1772 [running]:
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/flyteorg/flyteadmin/auth/authzserver.verifyClaims(0x23975a0|github.com/flyteorg/flyteadmin/auth/authzserver.verifyClaims(0x23975a0>?, 0xc001efb440)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/src/github.com/flyteorg/flyteadmin/auth/authzserver/provider.go:169 +0x7a6
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/flyteorg/flyteadmin/auth/authzserver.ResourceServer.ValidateAccessToken({{0x2988b00|github.com/flyteorg/flyteadmin/auth/authzserver.ResourceServer.ValidateAccessToken({{0x2988b00>, 0xc0001fee70}, {0xc000a691a0, 0x3, 0x3}}, {0x29ace40?, 0xc001d3d410?}, {0xc001bf2600, 0x20}, {0xc00097b507, ...})
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/src/github.com/flyteorg/flyteadmin/auth/authzserver/resource_server.go:46 +0x2df
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/flyteorg/flyteadmin/auth.GRPCGetIdentityFromAccessToken({0x29ace40|github.com/flyteorg/flyteadmin/auth.GRPCGetIdentityFromAccessToken({0x29ace40>, 0xc001d3d410}, {0x29b6ea0, 0xc000848120})
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/src/github.com/flyteorg/flyteadmin/auth/token.go:93 +0x1e5
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/flyteorg/flyteadmin/auth.GetAuthenticationInterceptor.func1({0x29ace40|github.com/flyteorg/flyteadmin/auth.GetAuthenticationInterceptor.func1({0x29ace40>, 0xc001d3d410})
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/src/github.com/flyteorg/flyteadmin/auth/handlers.go:242 +0xaf
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1({0x29ace40|github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1({0x29ace40>, 0xc001d3d410}, {0x2375fe0, 0xc001c7da40}, 0xc001c3e5e0, 0xc001c3e620)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/auth/auth.go:42 +0x93
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x29ace40|github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x29ace40>?, 0xc001d3d410?}, {0x2375fe0?, 0xc001c7da40?})
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:25 +0x3a
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/flyteorg/flyteadmin/auth.GetAuthenticationCustomMetadataInterceptor.func1({0x29ace40|github.com/flyteorg/flyteadmin/auth.GetAuthenticationCustomMetadataInterceptor.func1({0x29ace40>, 0xc001d3c300}, {0x2375fe0, 0xc001c7da40}, 0x20?, 0xc001c3e640)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/src/github.com/flyteorg/flyteadmin/auth/handlers.go:213 +0x323
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x29ace40|github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x29ace40>?, 0xc001d3c300?}, {0x2375fe0?, 0xc001c7da40?})
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:25 +0x3a
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x29ace40|github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x29ace40>, 0xc001d3c300}, {0x2375fe0, 0xc001c7da40}, 0x7f03ac8576d8?, 0xc001c3e660)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0x87
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x29ace40|github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x29ace40>?, 0xc001d3c300?}, {0x2375fe0?, 0xc001c7da40?})
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:25 +0x3a
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x29ace40|github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x29ace40>, 0xc001d3c300}, {0x2375fe0, 0xc001c7da40}, 0xc001a9eaf0?, 0x20f35e0?)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:34 +0xbf
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://github.com/flyteorg/flyteidl/gen/pb-go/flyteidl/service._AdminService_ListProjects_Handler({0x2444fa0|github.com/flyteorg/flyteidl/gen/pb-go/flyteidl/service._AdminService_ListProjects_Handler({0x2444fa0>?, 0xc001529000}, {0x29ace40, 0xc001d3c300}, 0xc001cfefc0, 0xc000e111d0)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/github.com/flyteorg/flyteidl@v1.1.5/gen/pb-go/flyteidl/service/admin.pb.go:1576 +0x138
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc000d1b180|google.golang.org/grpc.(*Server).processUnaryRPC(0xc000d1b180>, {0x29b5708, 0xc001a7f1e0}, 0xc000a5b0e0, 0xc000a4d200, 0x3c66e58, 0x0)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1283 +0xcfd
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://google.golang.org/grpc.(*Server).handleStream(0xc000d1b180|google.golang.org/grpc.(*Server).handleStream(0xc000d1b180>, {0x29b5708, 0xc001a7f1e0}, 0xc000a5b0e0, 0x0)
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1620 +0xa1b
    flyteadmin-b6b9c465c-vgj57 flyteadmin <http://google.golang.org/grpc.(*Server).serveStreams.func1.2()|google.golang.org/grpc.(*Server).serveStreams.func1.2()>
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:922 +0x98
    flyteadmin-b6b9c465c-vgj57 flyteadmin created by <http://google.golang.org/grpc.(*Server).serveStreams.func1|google.golang.org/grpc.(*Server).serveStreams.func1>
    flyteadmin-b6b9c465c-vgj57 flyteadmin 	/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:920 +0x28a
    I updated my
    allowedAudience
    to be as I was getting a wrong audience error (to match the audience in the token I am getting for flytectl)
    allowedAudience:
      - consoleClientId
      - propellerClientId
      - <api://flytectlClientId>
    p

    Prafulla Mahindrakar

    2 weeks ago
    scp is array of strings but seems in your case it returns a string when it parses the azure ad token. https://datatracker.ietf.org/doc/html/draft-ietf-oauth-token-exchange-06#section-4.2 can you check if this can be fixed in azure ad configuration or may be you can try adding a check in flyteadmin code if it can’t be typecasted to slice of strings then just populate it as single element scp list
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    yeah the claims are actually just a single string in this case
    p

    Prafulla Mahindrakar

    2 weeks ago
    Oh i think we just need a change here https://github.com/flyteorg/flyteadmin/blob/master/auth/authzserver/provider.go#L169 . Is that right since only the scopesClaim is not returning a list, though the key ScopeClaim exists
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    Yes the
    scp
    key exists, however it’s only a string instead of list of strings
    p

    Prafulla Mahindrakar

    2 weeks ago
    Created a image with the check ghcr.io/flyteorg/flyteadmin:v1.1.37-scp-fix . Do you want to try it . Needs to replaced in flyteadmin serve command image
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    Yes, I will try tonight as some teams are currently doing experiments on the Platform
    Thanks a lot Prafulla
    Here right,
    containers:
          - command:
            - flyteadmin
            - --config
            - /etc/flyte/config/*.yaml
            - serve
            image: <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.1.0|cr.flyte.org/flyteorg/flyteadmin-release:v1.1.0>
            imagePullPolicy: IfNotPresent
    p

    Prafulla Mahindrakar

    2 weeks ago
    yes
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    Admin is not failing anymore, but I am getting this error now, I think it is due to that the scopes that I am requesting is not matching the scopes in the JWT. Where is this checked?
    ~  flytectl get projects
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-07T06:59:26+02:00"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-07T06:59:26+02:00"}
    {"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-07T06:59:26+02:00"}
    {"json":{"src":"auth_flow_orchestrator.go:37"},"level":"debug","msg":"got a response from the refresh grant for old expiry 2022-09-07 08:23:35.302445 +0200 CEST with new expiry 2022-09-07 08:23:35.302445 +0200 CEST","ts":"2022-09-07T06:59:26+02:00"}
    {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-07T06:59:26+02:00"}
    Error: rpc error: code = Unauthenticated desc = authenticated user doesn't have required scope
    {"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unauthenticated desc = authenticated user doesn't have required scope","ts":"2022-09-07T06:59:26+02:00"}
    p

    Prafulla Mahindrakar

    2 weeks ago
    A scope named all is required in your authorization server https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server
    Under Scopes, click Add Scope. Set the name to all (required) and check Require user consent for this scope (recommended).
    Flyteadmin has a hard expectation to have this scope .https://github.com/flyteorg/flyteadmin/blob/master/auth/interceptor.go#L19
    You also would need to add that scope in here
    thirdPartyConfig:
                flyteClient:
                    # 3. Replace with a new Native/Public Client ID provisioned in the custom authorization server.
                    clientId: flytectl
    
                    # This should not change
                    redirectUri: <http://localhost:53593/callback>
    
                    # 4. "all" is a required scope and must be configured in the custom authorization server.
                    scopes:
                    - offline
                    - all
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    That works! Awesome
    If I don’t have a cookie already in my browser I get a timeout after like 20-30 seconds. Looks like the duration is set to two minutes in the pkce config https://github.com/flyteorg/flyteidl/blob/a93ef49d24aaac0b331035af0b15c76e16b4f4f4/clients/go/admin/config.go#L93
    But with a cookie already present it works like a charm
    p

    Prafulla Mahindrakar

    2 weeks ago
    Awesome . Finally we got this working .
    When you mean timeout after 20-30 secs , does the session timeout for you . because the new default is 2 minutes, but yeah thats not yet consumed in fllytectl . Just got merged a few hours ago https://github.com/flyteorg/flytectl/pull/350 The earlier default was 30 secs which is what you might be experiencing
    You can also pass this from commandline
    --admin.pkceConfig.timeout 2m
    A lot of hurdles before we got this running.thanks for sticking through the entire journey as this is flytes first azure ad setup and we learnt a lot of things along the way both for Azure ad and flyte auth. Would be great if you can help with documentation on this that can help the community. Thanks again. Cc: @Haytham Abuelfutuh
    Hampus Rosvall

    Hampus Rosvall

    2 weeks ago
    flytectl get projects --admin.pkceConfig.timeout 2m
    That works!
    Yes, I will make sure to write some proper docs. Quite busy with other stuff at work at the moment, but I will try to get started this week. I think in general it just boils down to poor docs from Azure side on how to configure the apps for the different use cases
    Also, many thanks for the assistance from Flyte team. Really awesome 🙂
    p

    Prafulla Mahindrakar

    2 weeks ago
    Sounds good. Thanks Hampus