Let s discuss auth

Question

< Hampus Rosvall> < Prafulla Mahindrakar> Let s discuss auth here

Haytham Abuelfutuh · Answer

Hampus Rosvall · Answer

I just checked the configmaps for propeller and scheduler there is a portion dedicated to client credentials auth flow but I can t seem to find an option to override the scopes here ```~ k get configmap flyte propeller scheduler config admin yaml | admin clientId clientId clientSecretLocation etc secrets client secret endpoint flyteadmin 81 insecure true```

Prafulla Mahindrakar · Answer

for eg for propeller For the above clientId we can define scope which is defined here

Prafulla Mahindrakar · Answer

You can add it to the same section with key scopes which takes a list in yaml

Hampus Rosvall · Answer

Right so something like ``` admin endpoint flyteadmin 81 insecure true clientId Values secrets adminOauthClientCredentials clientId clientSecretLocation etc secrets client secret scopes myScopes ```

Prafulla Mahindrakar · Answer

yes Or i guess yaml also uses this ```admin endpoint flyteadmin 81 insecure true clientId Values secrets adminOauthClientCredentials clientId clientSecretLocation etc secrets client secret scopes myscopes```

Hampus Rosvall · Answer

Cool let s try that

Prafulla Mahindrakar · Answer

BTW here the link to fast iteration that we spoke about <https docs flyte org projects cookbook en latest auto larger apps larger apps iterate html quickly re deploy your application>

Hampus Rosvall · Answer

Okay some progress I get sent to Azure IdP and get redirected with ```Cool Your authentication was successful and you can close the window ``` But I am getting this response now from the admin ```Error rpc error code = Unauthenticated desc = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken```

Hampus Rosvall · Answer

And in the logs from the admin it finds an existing metadata Bearer but when I validate the token on <http jwt io|jwt io> it says that the signature is invalid Maybe I have not configured the PKCE app successfully

Hampus Rosvall · Answer

Getting this error now <https github com AzureAD microsoft authentication library for js issues 2482>

Haytham Abuelfutuh · Answer

Hey < Hampus Rosvall> do you have time to debug this live now

Hampus Rosvall · Answer

Ah not by my computer atm

Hampus Rosvall · Answer

But I can try to dig into this more during the day tomorrow and we can have a call tomorrow if it s not resolved

Haytham Abuelfutuh · Answer

+1

Prafulla Mahindrakar · Answer

Hi < Hampus Rosvall> can you paste the logs from admin And also what are the config yaml for flytectl and your admin auth config If it easier we can get on a call to debug this

Hampus Rosvall · Answer

Hey sure By the way is there a way to override the allowed audience for the PKCE auth flow

Hampus Rosvall · Answer

``` appAuth authServerType External externalAuthServer baseUrl <https login microsoftonline com tenant v2 0> metadataUrl well known openid configuration allowedAudience clientid``` This is what I am doing for the client credentials flow and I am getting invalid audience error for the PKCE flow now

Hampus Rosvall · Answer

I think the error is within my app for PKCE flow and the scopes I am sending so I am just trying to figure out how it should be configured

Prafulla Mahindrakar · Answer

In your Azure authorization server can you check what is the audience configuration used The allowedAudience field needs to match one set in the authorization server

Hampus Rosvall · Answer

Yes when I validate the token the audience should equal the clientId

Hampus Rosvall · Answer

So I need to find a way to provide that during token validation as the allowed audience is different for client credentials flow and PKCE due to two different apps

Hampus Rosvall · Answer

<https github com flyteorg flyteadmin blob f72aa348e4a548d38f5afc5dcbc49f56fe67a75f auth authzserver provider go L138 L143> seems like the code checks if the audience from the claim is in the list of allowed audiences so I might be able to add it

Prafulla Mahindrakar · Answer

The allowed audience can be a list and its usually the endpoint which flyte is hosted on And this what should be configured in the allowed audience in admin so that token sent by auth server has the same audience configured in admin

Prafulla Mahindrakar · Answer

yes exactly

Prafulla Mahindrakar · Answer

For eg in okta this config is available directly on the auth server config page This not specific to an app but the capability of auth server when it rolls out token and who its intended audience is

Hampus Rosvall · Answer

```flyteadmin 76f6476884 9vnxq flyteadmin panic interface conversion interface is string not interface flyteadmin 76f6476884 9vnxq flyteadmin```

Hampus Rosvall · Answer

``` allowedAudience propellerClientId flytectlClientId```

Prafulla Mahindrakar · Answer

Can you double quote those values or use this representation ```key1 value1 value2 value3 value4 value5 ```

Prafulla Mahindrakar · Answer

Can we get on a call <https meet google com mqz nxot kta>

Hampus Rosvall · Answer

I can t right now have a meeting in 15 min

Prafulla Mahindrakar · Answer

Ok sure Let me know your availability

Hampus Rosvall · Answer

I have set up an app according to docs now I am getting the following error ```Tokens issued for the Single Page Application client type may only be redeemed via cross origin requests``` which seems to be discussed in this thread e g this solution Is this something I can add to the request

Prafulla Mahindrakar · Answer

Can you check if the failure comes after this <https github com flyteorg flyteidl blob master clients go admin pkce handle app call back go L44> And may be you would need to add the origin url when fetching the auth token at this place

Hampus Rosvall · Answer

Hm how would I check that The full error is from here ``` json src token source provider go 121 level error msg Error fetching token using auth flow due to error while exchanging auth code due to oauth2 cannot fetch token 400 Bad Request nResponse error invalid request error description Tokens issued for the Single Page Application client type may only be redeemed via cross origin requests```

Hampus Rosvall · Answer

<https github com flyteorg flyteidl blob 3ac7a0151ba11adab3116c083e85ceda9fee5c3d clients go admin pkce auth flow orchestrator go L128> we are getting to this line unclear how the callback is called in `serveMux`

Prafulla Mahindrakar · Answer

So after opening the url it redirects to auth server to login and then on the callback hits the app call back hander with authCode to be exchanged for the token The app call back handler exchanges that here So it seems in your case some policy blocks this exchange Also looking at this i guess its forbidden to set this Origin header And recommended way is to allowCors I remember we spoke about disabling this to test it out Is it still disabled in your admin config allowCors With allowCors do you get this far in the flow or does it fail earlier

Hampus Rosvall · Answer

This is the server portion of the `server yaml` configMap ``` server grpcPort 8089 httpPort 8088 security allowCors true allowedHeaders Content Type allowedOrigins secure false useAuth true```

Hampus Rosvall · Answer

So the error is still hit here ``` json src token source provider go 121 level error msg Error fetching token using auth flow due to error while exchanging auth code due to oauth2 cannot fetch token ```

Prafulla Mahindrakar · Answer

And what about the admin logs can we check those too for the error

Hampus Rosvall · Answer

```flyteadmin 858647d86d 8f4h8 flyteadmin json src handlers go 237 level debug msg Running authentication gRPC interceptor ts 2022 09 01T14 23 10Z flyteadmin 858647d86d 8f4h8 flyteadmin json src token go 83 level debug msg Could not retrieve bearer token from metadata rpc error code = Unauthenticated desc = Request unauthenticated with Bearer ts 2022 09 01T14 23 10Z flyteadmin 858647d86d 8f4h8 flyteadmin json src handlers go 247 level info msg Failed to parse Access Token from context Will attempt to find IDToken Error JWT VERIFICATION FAILED Could not retrieve bearer token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with Bearer ts 2022 09 01T14 23 10Z flyteadmin 858647d86d 8f4h8 flyteadmin json src token go 103 level debug msg Could not retrieve id token from metadata rpc error code = Unauthenticated desc = Request unauthenticated with IDToken ts 2022 09 01T14 23 10Z ```

Prafulla Mahindrakar · Answer

The message is clipped can you paste the entire log Is it the same issue ```Tokens issued for the Single Page Application client type may only be redeemed via cross origin requests```

Hampus Rosvall · Answer

``` json src token source provider go 121 level error msg Error fetching token using auth flow due to error while exchanging auth code due to oauth2 cannot fetch token 400 Bad Request nResponse error invalid request error description AADSTS9002327 Tokens issued for the Single Page Application client type may only be redeemed via cross origin requests json src client go 188 level warning msg Starting an unauthenticated client because error while exchanging auth code due to oauth2 cannot fetch token 400 Bad Request nResponse error invalid request error description AADSTS9002327 Tokens issued for the Single Page Application client type may only be redeemed via cross origin requests json src client go 64 level info msg Initialized Admin client ts 2022 09 01T16 23 09+02 00 Error rpc error code = Unauthenticated desc = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken json src main go 13 level error msg rpc error code = Unauthenticated desc = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken ts 2022 09 01T16 23 10+02 00 ```

Hampus Rosvall · Answer

The dots are just error codes any timestamps from Azure

Hampus Rosvall · Answer

I am requesting the openid scope btw

Prafulla Mahindrakar · Answer

< Hampus Rosvall> I think the app shouldn t be an SPA but a native app This is how we use it for Okta

Prafulla Mahindrakar · Answer

This fits well with flytectl usecase Browserless app <https docs microsoft com en us azure active directory develop scenario desktop overview>

Hampus Rosvall · Answer

Ah awesome will try that out Thanks a lot

Hampus Rosvall · Answer

When I use a mobile native app I am actually getting to this line here i e the browser says I authenticated correctly Here are the logs from the admin ```flyteadmin 858647d86d mxzv5 flyteadmin json src handlers go 209 level debug msg Found existing metadata Bearer JWT TOKEN ts 2022 09 04T06 42 05Z flyteadmin 858647d86d mxzv5 flyteadmin json src handlers go 237 level debug msg Running authentication gRPC interceptor ts 2022 09 04T06 42 05Z flyteadmin 858647d86d mxzv5 flyteadmin json src handlers go 247 level info msg Failed to parse Access Token from context Will attempt to find IDToken Error failed to verify id token signature ts 2022 09 04T06 42 05Z flyteadmin 858647d86d mxzv5 flyteadmin json src token go 103 level debug msg Could not retrieve id token from metadata rpc error code = Unauthenticated desc = Request unauthenticated with IDToken ts 2022 09 04T06 42 05Z ``` Here is the problem most likely ```flyteadmin 858647d86d mxzv5 flyteadmin json src handlers go 247 level info msg Failed to parse Access Token from context Will attempt to find IDToken Error failed to verify id token signature ts 2022 09 04T06 38 59Z ``` After I am getting the successful auth message i e this <https github com flyteorg flyteidl blob master clients go admin pkce handle app call back go L52> in the browser I am seeing these logs in from flytectl ```Error rpc error code = Unauthenticated desc = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken json src main go 13 level error msg rpc error code = Unauthenticated desc = token parse error JWT VERIFICATION FAILED Could not retrieve id token from metadata caused by rpc error code = Unauthenticated desc = Request unauthenticated with IDToken ts 2022 09 04T08 42 05+02 00 ```

Hampus Rosvall · Answer

In the token header I am getting a `nonce` parameter seems like OpenID Connect nonce and PKCE are not entirely the same auth flow as to where the token validation takes place

Prafulla Mahindrakar · Answer

Ok that one good news So the auth flow completes and generated the token but the token is rejected by admin while verifying the signature Nonce is part of the header So i am not sure if that causing an issue though Unfortunately the same error is printed at two places we will have to check both Assuming first time nothing is in the cache so it should fail at the L140 Can you verify that if you open baseUrl metadataUrl that you have property called jwks uri jwksUri Open the url pointed to jwks uri jwksUri And note the kid property in the keys returned There could be multiple keys Can you try pasting your saved token and use the access token from it and verify it on Does the signature show its verified over there Also note the kid in the header You can get kid also by running this code Also paste the token in here and can you check the KeyID printed Does it match the KID from jwks uri data

Hampus Rosvall · Answer

Yeah the KID in the token matches the KID in the jwks uri However the token says invalid signature when I try to validate it

Hampus Rosvall · Answer

Hampus Rosvall · Answer

I think this has something to do with it

Prafulla Mahindrakar · Answer

Thanks for checking that Lot of discussion and many ways folks have tried diff ways to resolve it Audience and scope configuration can also lead to this looks like Can you check what is the audience configured on the Azure ad auth server In case of okta its the flyte endpoint Also the scope may be introduce a new scope for the app detailed here < Haytham Abuelfutuh> can you suggest what else should we try here

Prafulla Mahindrakar · Answer

From the comments `special processing` from what i get means only the resource server in our case flyteadmin who is entitled to receive the token can understand the data

Prafulla Mahindrakar · Answer

This err which is not printed anywhere may be can give us more clue <https github com coreos go oidc blob 8d771559cf6e5111c9b9159810d0e4538e7cdc82 jwks go L135> You can try adding a log statement and consuming it in flyteadmin

Hampus Rosvall · Answer

I followed the steps from the github issue comment and I am actually getting a valid token now but the admin fails here after finding a bearer token I remember us having this issue previously but I am not sure how we solved it ```flyteadmin b6b9c465c vgj57 flyteadmin json src handlers go 237 level debug msg Running authentication gRPC interceptor ts 2022 09 06T06 15 34Z flyteadmin b6b9c465c vgj57 flyteadmin panic interface conversion interface is string not interface flyteadmin b6b9c465c vgj57 flyteadmin flyteadmin b6b9c465c vgj57 flyteadmin goroutine 1772 running flyteadmin b6b9c465c vgj57 flyteadmin 0xc001efb440 flyteadmin b6b9c465c vgj57 flyteadmin go src github com flyteorg flyteadmin auth authzserver provider go 169 +0x7a6 flyteadmin b6b9c465c vgj57 flyteadmin 0xc0001fee70 0xc000a691a0 0x3 0x3 0x29ace40 0xc001d3d410 0xc001bf2600 0x20 0xc00097b507 flyteadmin b6b9c465c vgj57 flyteadmin go src github com flyteorg flyteadmin auth authzserver resource server go 46 +0x2df flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3d410 0x29b6ea0 0xc000848120 flyteadmin b6b9c465c vgj57 flyteadmin go src github com flyteorg flyteadmin auth token go 93 +0x1e5 flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3d410 flyteadmin b6b9c465c vgj57 flyteadmin go src github com flyteorg flyteadmin auth handlers go 242 +0xaf flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3d410 0x2375fe0 0xc001c7da40 0xc001c3e5e0 0xc001c3e620 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com grpc ecosystem go grpc middleware v1 2 2 auth auth go 42 +0x93 flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3d410 0x2375fe0 0xc001c7da40 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com grpc ecosystem go grpc middleware v1 2 2 chain go 25 +0x3a flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3c300 0x2375fe0 0xc001c7da40 0x20 0xc001c3e640 flyteadmin b6b9c465c vgj57 flyteadmin go src github com flyteorg flyteadmin auth handlers go 213 +0x323 flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3c300 0x2375fe0 0xc001c7da40 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com grpc ecosystem go grpc middleware v1 2 2 chain go 25 +0x3a flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3c300 0x2375fe0 0xc001c7da40 0x7f03ac8576d8 0xc001c3e660 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com grpc ecosystem go grpc prometheus v1 2 0 server metrics go 107 +0x87 flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3c300 0x2375fe0 0xc001c7da40 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com grpc ecosystem go grpc middleware v1 2 2 chain go 25 +0x3a flyteadmin b6b9c465c vgj57 flyteadmin 0xc001d3c300 0x2375fe0 0xc001c7da40 0xc001a9eaf0 0x20f35e0 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com grpc ecosystem go grpc middleware v1 2 2 chain go 34 +0xbf flyteadmin b6b9c465c vgj57 flyteadmin 0xc001529000 0x29ace40 0xc001d3c300 0xc001cfefc0 0xc000e111d0 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod github com flyteorg flyteidl v1 1 5 gen pb go flyteidl service admin pb go 1576 +0x138 flyteadmin b6b9c465c vgj57 flyteadmin 0x29b5708 0xc001a7f1e0 0xc000a5b0e0 0xc000a4d200 0x3c66e58 0x0 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod google golang org grpc v1 46 0 server go 1283 +0xcfd flyteadmin b6b9c465c vgj57 flyteadmin 0x29b5708 0xc001a7f1e0 0xc000a5b0e0 0x0 flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod google golang org grpc v1 46 0 server go 1620 +0xa1b flyteadmin b6b9c465c vgj57 flyteadmin flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod google golang org grpc v1 46 0 server go 922 +0x98 flyteadmin b6b9c465c vgj57 flyteadmin created by flyteadmin b6b9c465c vgj57 flyteadmin go pkg mod google golang org grpc v1 46 0 server go 920 +0x28a```

Hampus Rosvall · Answer

I updated my `allowedAudience` to be as I was getting a wrong audience error to match the audience in the token I am getting for flytectl ```allowedAudience consoleClientId propellerClientId <api flytectlClientId>```

Prafulla Mahindrakar · Answer

scp is array of strings but seems in your case it returns a string when it parses the azure ad token <https datatracker ietf org doc html draft ietf oauth token exchange 06 section 4 2> can you check if this can be fixed in azure ad configuration or may be you can try adding a check in flyteadmin code if it can t be typecasted to slice of strings then just populate it as single element scp list

Hampus Rosvall · Answer

yeah the claims are actually just a single string in this case

Hampus Rosvall · Answer

<https github com flyteorg flyteadmin blob b29605d86cfe68f3b8a1519c10fd971f4251cfae auth authzserver provider go L130> so we need to add check here

Prafulla Mahindrakar · Answer

Oh i think we just need a change here <https github com flyteorg flyteadmin blob master auth authzserver provider go L169> Is that right since only the scopesClaim is not returning a list though the key ScopeClaim exists

Hampus Rosvall · Answer

Yes the `scp` key exists however it s only a string instead of list of strings

Prafulla Mahindrakar · Answer

Created a image with the check <http ghcr io flyteorg flyteadmin v1 1 37 scp fix|ghcr io flyteorg flyteadmin v1 1 37 scp fix> Do you want to try it Needs to replaced in flyteadmin serve command image

Hampus Rosvall · Answer

Yes I will try tonight as some teams are currently doing experiments on the Platform

Hampus Rosvall · Answer

Thanks a lot Prafulla

Hampus Rosvall · Answer

Here right ``` containers command flyteadmin config etc flyte config yaml serve image <http cr flyte org flyteorg flyteadmin release v1 1 0|cr flyte org flyteorg flyteadmin release v1 1 0> imagePullPolicy IfNotPresent```

Prafulla Mahindrakar · Answer

yes

Hampus Rosvall · Answer

Admin is not failing anymore but I am getting this error now I think it is due to that the scopes that I am requesting is not matching the scopes in the JWT Where is this checked ``` ~ flytectl get projects json src viper go 398 level debug msg Config section storage updated No update handler registered ts 2022 09 07T06 59 26+02 00 json src viper go 398 level debug msg Config section root updated No update handler registered ts 2022 09 07T06 59 26+02 00 json src viper go 400 level debug msg Config section admin updated Firing updated event ts 2022 09 07T06 59 26+02 00 json src auth flow orchestrator go 37 level debug msg got a response from the refresh grant for old expiry 2022 09 07 08 23 35 302445 +0200 CEST with new expiry 2022 09 07 08 23 35 302445 +0200 CEST ts 2022 09 07T06 59 26+02 00 json src client go 64 level info msg Initialized Admin client ts 2022 09 07T06 59 26+02 00 Error rpc error code = Unauthenticated desc = authenticated user doesn t have required scope json src main go 13 level error msg rpc error code = Unauthenticated desc = authenticated user doesn t have required scope ts 2022 09 07T06 59 26+02 00 ```

Prafulla Mahindrakar · Answer

A scope named all is required in your authorization server <https docs flyte org en latest deployment cluster config auth setup html oauth2 authorization server> ```Under Scopes click Add Scope Set the name to all required and check Require user consent for this scope recommended ```

Prafulla Mahindrakar · Answer

Flyteadmin has a hard expectation to have this scope <https github com flyteorg flyteadmin blob master auth interceptor go L19>

Prafulla Mahindrakar · Answer

You also would need to add that scope in here ``` thirdPartyConfig flyteClient 3 Replace with a new Native Public Client ID provisioned in the custom authorization server clientId flytectl This should not change redirectUri <http localhost 53593 callback> 4 all is a required scope and must be configured in the custom authorization server scopes offline all```

Hampus Rosvall · Answer

That works Awesome

Hampus Rosvall · Answer

If I don t have a cookie already in my browser I get a timeout after like 20 30 seconds Looks like the duration is set to two minutes in the pkce config <https github com flyteorg flyteidl blob a93ef49d24aaac0b331035af0b15c76e16b4f4f4 clients go admin config go L93>

Hampus Rosvall · Answer

But with a cookie already present it works like a charm

Prafulla Mahindrakar · Answer

Awesome Finally we got this working

Prafulla Mahindrakar · Answer

When you mean timeout after 20 30 secs does the session timeout for you because the new default is 2 minutes but yeah thats not yet consumed in fllytectl Just got merged a few hours ago <https github com flyteorg flytectl pull 350> The earlier default was 30 secs which is what you might be experiencing

Prafulla Mahindrakar · Answer

You can also pass this from commandline ` admin pkceConfig timeout 2m`

Prafulla Mahindrakar · Answer

A lot of hurdles before we got this running thanks for sticking through the entire journey as this is flytes first azure ad setup and we learnt a lot of things along the way both for Azure ad and flyte auth Would be great if you can help with documentation on this that can help the community Thanks again Cc < Haytham Abuelfutuh>

Hampus Rosvall · Answer

```flytectl get projects admin pkceConfig timeout 2m```

Hampus Rosvall · Answer

That works

Hampus Rosvall · Answer

Yes I will make sure to write some proper docs Quite busy with other stuff at work at the moment but I will try to get started this week I think in general it just boils down to poor docs from Azure side on how to configure the apps for the different use cases

Hampus Rosvall · Answer

Also many thanks for the assistance from Flyte team Really awesome slightly smiling face

Prafulla Mahindrakar · Answer

Sounds good Thanks Hampus