Hey, I am setting up OIDC Authentication using Azure AD IdP. I followed the steps here. Right now when I hit the console I am prompted to authenticate against our IdP, then I get sent back to the callback URI,

https://<host>/callback

and get stuck in some redirect loop where the

FlyteAdmin

Pod keeps logging like below. Does anyone know what could cause this? Thanks 🙂

flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie_manager.go:71"},"level":"info","msg":"Refresh token doesn't exist or failed to read it. Ignoring this error. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_rt], caused by: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:227"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:64"},"level":"debug","msg":"JWT parsing with claims failed failed to verify signature: failed to verify id token signature","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}

Hi @helpful-crowd-74546, can you verify the JWKSUri is configured correctly in the Azure OIDC . Seems its unable to verify the token signature

failed to verify id token signature

and it could potentially point to a misconfigured JWKS URI

Right, doing some investigation! Thanks 🙂

we dont have an example of Azure AD unfortunately , but i believe @sticky-printer-535 and his team from the community have setup flyte on Azure with auth and might have some info on this. Also adding @high-park-82026 on this thread

Here is the jkws_uri from the open id configuartion, was there anything else I should provide you? 🙂

"jwks_uri": "<https://login.microsoftonline.com/><tenant_id>/discovery/v2.0/keys"

Did you try baseUrl with

<https://login.microsoftonline.com/{tenant}/v2.0>

Yes, that is what we are doing now. https://host/me now yields

"token parse error [JWT_VERIFICATION_FAILED] jwt parse with claims failed, caused by: failed to verify signature: failed to verify id token signature"

Can you try using some postman type client and retrieve the auth token and verify that the key id in the header which you can get using https://jwt.io/ Is the same as show in

<ttps://login.microsoftonline.com/><tenant_id>/discovery/v2.0/keys

@helpful-crowd-74546 I spent some time on Flyte <-> Azure, but I focused mostly on the storage side of it. There are some pitfalls with fsspec that you will run into. Im am happy to help there. Unfortunately, I did not look into authentication yet.

Right, I will look into what you sent @icy-agent-73298, thanks! @sticky-printer-535 basically we are only using Azure for the Active Directory, we are deploying on EKS 🙂

Oh that’s a great find! Thank you for digging into that @helpful-crowd-74546! I think we can make

/.well-known/openid-configuration

value a config option with

/.well-known/openid-configuration

as the default… and in your deployment you can customize it to append the app id.. what do you think?

@helpful-crowd-74546 are you folks on AWS or azure?

@high-park-82026 @helpful-crowd-74546 the wellknown endpoint is configurable in flyteadmin through metadataUrl eg :

baseUrl: <https://login.microsoftonline.com/{tenant}/v2.0>
 metadataUrl: .well-known/openid-configuration?appid=6731de76-14a6-49ae-97bc-6eba6914391e

This should avoid any code changes specific tp azure

Hi again, I did some more reading on this matter and it might not be as trivial as I first thought. In this article it is stated that only apps registered with claims-mapping require to append

app_id=<clientId>

to the issuer URL. Hence, it could be that it works out of the box with Azure AD IdP given an app registered without customized claims (please see here for more info) - not sure at this point if claims are customized on a company level or per app, but we should try this. Otherwise it seems to me that we would need to infer in the ParseIDTokenAndValidate: 1. If the issuer is Microsoft and the Azure AD app is registered 2. Then add

?app_id=<clientId>

to JWTURL However, given the scope of this function, the information about the JWTURL is not available as far as I can tell so we would need to pass that along. What you suggest @User sounds good, i.e., initialize the Provider using the ProviderConfig struct where the

Provider.remoteKeySet

is created given some logic to infer whether the Issuer is customized claims Azure AD app. This alternative however requires to bump the oidc package from v2.x to v3.x as this feature was added only in v3 it seems. I have very little experience with authentication and different identitiy providers so I could of course be missing something! What do you guys think is the best way forward here?

Hi @helpful-crowd-74546 thanks for sharing this info. Seems claim-mapping allows more customization within an orgianization(tenant) and requiring that seems the discovery url needs to change to have the appId suffix. The RFC for open id mentions the discovery to be available at /.well-known/openid-configuration https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfig which is what the go-oidc has implemented . • We can pursue the path of asking the go-oidc members to have this configurable and through which flyteadmin can use this configurable option. But this might take a while but seems be most non intrusive and less maintenance in the long run. • Short term we can try without the claim-mapping which would be using the standard openid config for azure and that i believe should run without issue. Can we try this first as you mentioned. • Other option is to maintain the ProviderConfig in the flyteadmin auth config. And use that in ParseIdTokenAndValidate function. During the creation of the provider it would use it from the config . cc : @high-park-82026 . Bumping up the oidc package should be fine but want to be sure we want to take that config in the admin but if go-oidc doesn’t want to take that change then yeh we would have to maintain it in flyteadmin

Yeah, I will try to experiment with an app registered without custom claims on Monday to see if the OIDC authentication works out of the box given those circumstances. Otherwise I think we can figure out something along with bullet point three as a short term solution, or just have a conditional where we check if the provider is Microsoft then we append modify the well known end point in order to extract the correct

ProviderConfig

. I suggest we wait until Monday when I hopefully get to experiment with a new Azure app, and we decide on next steps after that. Sounds good?

Sure sounds good @helpful-crowd-74546.

I need help from another department to hook up a new Azure AD app, will update once they have the time to do so..

Ok 👍

Final update. I registered a new app in Azure AD without custom claims and it works out of the box without any hacky modifications in the JWT token verifier. I can update the docs w.r.t. Azure AD oidc authentication. Can you please point me to where you are documenting this in your source code @icy-agent-73298 🙂

Thats great news @helpful-crowd-74546 🦜. The oidc docs are available here https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#auth-setup and the code corr to it resides here https://github.com/flyteorg/flyte/blob/master/rsts/deployment/cluster_config/auth_setup.rst Similar to how we have added caveats for keycloak and okta, you can add one for azure AD. Also you can add source code level docs on the ParseIdTokenAndValidate which would show up here https://pkg.go.dev/github.com/flyteorg/flyteadmin/auth@v0.6.138#ParseIDTokenAndValidate

Thank you @helpful-crowd-74546