Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Hey, I am setting up OIDC Authentication using Azure AD IdP. I followed the steps here. Right now when I hit the console I am prompted to authenticate against our IdP, then I get sent back to the callback URI,
    https://<host>/callback
    and get stuck in some redirect loop where the
    FlyteAdmin
    Pod keeps logging like below. Does anyone know what could cause this? Thanks 🙂
    flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie_manager.go:71"},"level":"info","msg":"Refresh token doesn't exist or failed to read it. Ignoring this error. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_rt], caused by: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:227"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:64"},"level":"debug","msg":"JWT parsing with claims failed failed to verify signature: failed to verify id token signature","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
    p

    Prafulla Mahindrakar

    5 months ago
    Hi @Hampus Rosvall, can you verify the JWKSUri is configured correctly in the Azure OIDC . Seems its unable to verify the token signature
    failed to verify id token signature
    and it could potentially point to a misconfigured JWKS URI
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Right, doing some investigation! Thanks 🙂
    So checking the callstack in the developer console in the browser we are authenticating properly on Azure AD, setting the cookies and we are redirected back to the console, where the get stuck in a loop. Could of course still be some error in the AD but at first glance it looks correct. Hitting the endpoint
    https://<host>/me
    reproduces the error we see in the FlyteAdmin logs. I am probably missing something, but it is hard to defer what exactly, do you have any examples of Azure AD to make sure we are using the correct baseUrl for instance? We are currently using
    baseUrl: <https://sts.windows.net/><tentant_id>/
    and have also tried `baseUrl: https://login.microsoft.net/<tenant_id>/`with similar issues
    p

    Prafulla Mahindrakar

    5 months ago
    we dont have an example of Azure AD unfortunately , but i believe @Stephan Gref and his team from the community have setup flyte on Azure with auth and might have some info on this. Also adding @Haytham Abuelfutuh on this thread
    Can you check open id configuration of your app here
    <https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration?appId=>.....
    Referred this doc https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-protocols-oidc Can we check the jwks_uri . Probably provide the o/p of this metadata might help
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Here is the jkws_uri from the open id configuartion, was there anything else I should provide you? 🙂
    "jwks_uri": "<https://login.microsoftonline.com/><tenant_id>/discovery/v2.0/keys"
    p

    Prafulla Mahindrakar

    5 months ago
    Did you try baseUrl with
    <https://login.microsoftonline.com/{tenant}/v2.0>
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Yes, that is what we are doing now. https://host/me now yields
    "token parse error [JWT_VERIFICATION_FAILED] jwt parse with claims failed, caused by: failed to verify signature: failed to verify id token signature"
    And the admin finds the cookies it seems like
    flyteadmin-5df48d7fc-5pjql flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-06T11:45:48Z"}
    flyteadmin-5df48d7fc-5pjql flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-06T11:45:48Z"}
    flyteadmin-5df48d7fc-5pjql flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_rt] cookie found","ts":"2022-04-06T11:45:48Z"}
    flyteadmin-5df48d7fc-5pjql flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-04-06T11:45:48Z"}
    p

    Prafulla Mahindrakar

    5 months ago
    Can you try using some postman type client and retrieve the auth token and verify that the key id in the header which you can get using https://jwt.io/ Is the same as show in
    <ttps://login.microsoftonline.com/><tenant_id>/discovery/v2.0/keys
    I followed this URL to generate a token for my keycloak OIDC client installation https://sis-cc.gitlab.io/dotstatsuite-documentation/configurations/authentication/token-in-postman/ But it should be similar for any other OIDC provider
    Stephan Gref

    Stephan Gref

    5 months ago
    @Hampus Rosvall I spent some time on Flyte <-> Azure, but I focused mostly on the storage side of it. There are some pitfalls with fsspec that you will run into. Im am happy to help there. Unfortunately, I did not look into authentication yet.
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Right, I will look into what you sent @Prafulla Mahindrakar, thanks! @Stephan Gref basically we are only using Azure for the Active Directory, we are deploying on EKS 🙂
    Update! @Prafulla Mahindrakar The key id in the JWT token matches
    <https://login.microsoftonline.com/><tenant_id>/discovery/v2.0/keys?appId=<app_id>
    . Note the additional
    appId
    parameter needed for Azure AD apps with custom signing keys. This is discussed in this issue here (originating from https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-protocols-oidc#sample-response)
    Another update. Updating the ID Token verification with respect to what is stated in the Issue works! I will draft a PR with some suggested changes for Azure AD IdP 🙂
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    5 months ago
    Oh that’s a great find! Thank you for digging into that @Hampus Rosvall! I think we can make
    /.well-known/openid-configuration
    value a config option with
    /.well-known/openid-configuration
    as the default… and in your deployment you can customize it to append the app id.. what do you think?
    CC @Ketan (kumare3) as he’s interested
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    @Hampus Rosvall are you folks on AWS or azure?
    last I helped you run it on AWS 😄
    p

    Prafulla Mahindrakar

    5 months ago
    @Haytham Abuelfutuh @Hampus Rosvall the wellknown endpoint is configurable in flyteadmin through metadataUrl eg :
    baseUrl: <https://login.microsoftonline.com/{tenant}/v2.0>
     metadataUrl: .well-known/openid-configuration?appid=6731de76-14a6-49ae-97bc-6eba6914391e
    This should avoid any code changes specific tp azure
    I take that back. This is only available as option in external auth server (https://github.com/flyteorg/flyteadmin/blob/5993c4e09d05faae12a21bb8b3e536f488bc0d6e/auth/config/config.go#L185) config and not for openId admin (https://github.com/flyteorg/flyteadmin/blob/5993c4e09d05faae12a21bb8b3e536f488bc0d6e/auth/config/config.go#L233) config yet . Let us know @Hampus Rosvall if you will be interested in making this change
    @Haytham Abuelfutuh i think in the oidc case it cannot be configured since the oidc.NewProvider func hardcodes the well known url and we need one with appended appId. The workaround suggested in the linked issue is what Hampus tried and worked for him . Another way would be to initialize the provider completely like here https://github.com/coreos/go-oidc/blob/2d47dd95152744f41009de5797efe6cc07832a41/oidc/oidc.go#L168 Fixing oidc to paremterize the well know url would be much cleaner. What would you suggest here.
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Hi again, I did some more reading on this matter and it might not be as trivial as I first thought. In this article it is stated that only apps registered with claims-mapping require to append
    app_id=<clientId>
    to the issuer URL. Hence, it could be that it works out of the box with Azure AD IdP given an app registered without customized claims (please see here for more info) - not sure at this point if claims are customized on a company level or per app, but we should try this. Otherwise it seems to me that we would need to infer in the ParseIDTokenAndValidate:1. If the issuer is Microsoft and the Azure AD app is registered 2. Then add
    ?app_id=<clientId>
    to JWTURL However, given the scope of this function, the information about the JWTURL is not available as far as I can tell so we would need to pass that along. What you suggest @User sounds good, i.e., initialize the Provider using the ProviderConfig struct where the
    Provider.remoteKeySet
    is created given some logic to infer whether the Issuer is customized claims Azure AD app. This alternative however requires to bump the oidc package from v2.x to v3.x as this feature was added only in v3 it seems. I have very little experience with authentication and different identitiy providers so I could of course be missing something! What do you guys think is the best way forward here?
    p

    Prafulla Mahindrakar

    5 months ago
    Hi @Hampus Rosvall thanks for sharing this info. Seems claim-mapping allows more customization within an orgianization(tenant) and requiring that seems the discovery url needs to change to have the appId suffix. The RFC for open id mentions the discovery to be available at /.well-known/openid-configuration https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfig which is what the go-oidc has implemented . • We can pursue the path of asking the go-oidc members to have this configurable and through which flyteadmin can use this configurable option. But this might take a while but seems be most non intrusive and less maintenance in the long run. • Short term we can try without the claim-mapping which would be using the standard openid config for azure and that i believe should run without issue. Can we try this first as you mentioned. • Other option is to maintain the ProviderConfig in the flyteadmin auth config. And use that in ParseIdTokenAndValidate function. During the creation of the provider it would use it from the config . cc : @Haytham Abuelfutuh . Bumping up the oidc package should be fine but want to be sure we want to take that config in the admin but if go-oidc doesn’t want to take that change then yeh we would have to maintain it in flyteadmin
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Yeah, I will try to experiment with an app registered without custom claims on Monday to see if the OIDC authentication works out of the box given those circumstances. Otherwise I think we can figure out something along with bullet point three as a short term solution, or just have a conditional where we check if the provider is Microsoft then we append modify the well known end point in order to extract the correct
    ProviderConfig
    . I suggest we wait until Monday when I hopefully get to experiment with a new Azure app, and we decide on next steps after that. Sounds good?
    p

    Prafulla Mahindrakar

    5 months ago
    Sure sounds good @Hampus Rosvall.
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    I need help from another department to hook up a new Azure AD app, will update once they have the time to do so..
    p

    Prafulla Mahindrakar

    5 months ago
    Ok 👍
    Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Final update. I registered a new app in Azure AD without custom claims and it works out of the box without any hacky modifications in the JWT token verifier. I can update the docs w.r.t. Azure AD oidc authentication. Can you please point me to where you are documenting this in your source code @Prafulla Mahindrakar 🙂
    p

    Prafulla Mahindrakar

    5 months ago
    Thats great news @Hampus Rosvall 😛artyparrot:. The oidc docs are available here https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#auth-setup and the code corr to it resides here https://github.com/flyteorg/flyte/blob/master/rsts/deployment/cluster_config/auth_setup.rst Similar to how we have added caveats for keycloak and okta, you can add one for azure AD. Also you can add source code level docs on the ParseIdTokenAndValidate which would show up here https://pkg.go.dev/github.com/flyteorg/flyteadmin/auth@v0.6.138#ParseIDTokenAndValidate
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    Thank you @Hampus Rosvall