https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
r

Rio

05/22/2023, 7:38 PM
đź‘‹ I'm running into an auth problem with flytectl using the ClientSecret (client credentials) method. I've assigned the proper scopes to the oauth2 client and the flyteadmin service isn't complaining about not being able to find the token in the metadata anymore which is nice. However now I'm getting a response to flytectl that I'm missing a scope:
Copy code
Error: Connection Info: [Endpoint: dns:///flyte.localhost:80, InsecureConnection?: true, AuthMode: ClientSecret]: rpc error: code = Unauthenticated desc = authenticated user doesn't have required scope
LinkerD confirms that it's the flyteadmin container that responds with Unauthenticated. This is quite perplexing and I've verified that the token is sent and it contains all the scopes required: 
"scope": "all email offline profile"
I hope people can give me some hints as I've been bashing my head for way to long against this. I'm just trying to have a flytectl register stuff in CI.
Just for completeness, here's an example of the token that get's sent in the 
flyte-authorization
header:
Copy code
{
  "exp": 1684787333,
  "iat": 1684787033,
  "jti": "984df155-c8fc-41f4-9e33-65e38eb51e06",
  "iss": "<http://keycloak.localhost/realms/flyte>",
  "aud": "account",
  "sub": "02bf7fe8-1b4a-4b5f-a08c-378eb26be405",
  "typ": "Bearer",
  "azp": "flytectl",
  "session_state": "43678964-d918-4557-b9ae-ce3639e8cfe3",
  "acr": "1",
  "allowed-origins": [
    "/*"
  ],
  "realm_access": {
    "roles": [
      "default-roles-flyte",
      "offline_access",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid profile email all offline",
  "sid": "43678964-d918-4557-b9ae-ce3639e8cfe3",
  "email_verified": false,
  "clientHost": "10.42.0.50",
  "preferred_username": "service-account-flytectl",
  "clientAddress": "10.42.0.50",
  "client_id": "flytectl"
}
This matches the access_token that's returned from keycloak to flytectl. However, if I enable openid as a scope in the auth request, it also returns an id_token which looks like the following. Note the lack of scopes here.
Copy code
{
  "exp": 1684787117,
  "iat": 1684786817,
  "auth_time": 0,
  "jti": "07555916-cb38-485f-ae29-e4af0a183dfb",
  "iss": "<http://keycloak.localhost/realms/flyte>",
  "aud": "flytectl",
  "sub": "02bf7fe8-1b4a-4b5f-a08c-378eb26be405",
  "typ": "ID",
  "azp": "flytectl",
  "session_state": "f6c36f22-0614-4e58-811f-0ceb02916037",
  "at_hash": "BqKQDMlf3921-RZSmzaHtA",
  "acr": "1",
  "sid": "f6c36f22-0614-4e58-811f-0ceb02916037",
  "email_verified": false,
  "clientHost": "10.42.0.50",
  "preferred_username": "service-account-flytectl",
  "clientAddress": "10.42.0.50",
  "client_id": "flytectl"
}
Seeing as flyteadmin also does a request towards keycloak as shown by this linkerd screenshot, is it possible it uses the access_token to get more information about the service account and then fails when that information does not actually return scopes?
s

Samhita Alla

05/23/2023, 5:23 AM
Have you set 
insecure
to true in your flytectl config?
r

Rio

05/23/2023, 6:38 AM
Yes i have. Along with the ClientSecret authtype and clientId that matches my oauth2 app.
s

Samhita Alla

05/23/2023, 8:28 AM
Can you set it to false?
r

Rio

05/23/2023, 8:49 AM
I can but it won't work as this POC setup has no tls enabled. What were you expecting it to do?
As expected I'm getting transport connection errors now.
is there some hidden extra auth logic that gets triggered when we change transport layer settings?
s

Samhita Alla

05/23/2023, 9:20 AM
r

Rio

05/23/2023, 9:35 AM
Ah no biggie
I do have the scopes all and offline enabled and assigned in keycloak along with adding it to the local flytectl config.
I've read the thread twice now and it's not helping me. As I've mentioned before I've verified that the 
flyte-authentication
header contains 
Bearer <token>
which, when decoded, contains the proper scopes.
I can't believe it's just me... Isn't anyone using a CI system to talk to flyte? Or has everyone simply disabled auth all together? Maybe using an oauth proxy infront of it then?
k

Ketan (kumare3)

05/23/2023, 12:54 PM
You cannot use auth without TLS. That’s is incredibly dangerous and susceptible to things like man in the middle attacks, spoofing, eavesdropping etc. Grpc also won’t allow it. Can you bind a cert
r

Rio

05/23/2023, 1:04 PM
I understand it's dangerous but as I said before this is a POC local setup. gRPC actually does allow it, otherwise the system as it is now wouldn't work as lots of internal calls and flytectl calls are actually gRPC.
k

Ketan (kumare3)

05/23/2023, 1:23 PM
Are you saying flytectl works?
r

Rio

05/23/2023, 1:26 PM
Not when i enable auth with client credentials, otherwise it works. I was talking about gRPC your remark of gRPC not allowing calls without TLS, which seems orthogonal to the scope problem.
k

Ketan (kumare3)

05/23/2023, 1:27 PM
Grpc without TLS and auth works, but enabling auth and no TLS won’t work
r

Rio

05/23/2023, 1:30 PM
I'm not getting transport related errors though, just a missing scope error. So you're saying there's some undocumented logic that toggling TLS at the transport layer influences application layer authentication?
k

Ketan (kumare3)

05/23/2023, 1:36 PM
Why are you trying to run auth in sandbox? You agree that in real setting you will have TLS and only then enable auth
r

Rio

05/23/2023, 2:06 PM
Why are you trying to run auth in sandbox?
I'm not running the sandbox setup as described in the docs, but the single cluster production setup. This is an exploratory experiment to find all the component behaviors and failure modes before shaping it into more hardened environments.
You agree that in real setting you will have TLS and only then enable auth
Yes but only for user auth probably, not service account based activity like 
flytectl register
as these processes most likely will run in the same cluster and will be covered by mTLS and network policies.
k

Ketan (kumare3)

05/23/2023, 2:22 PM
Cc @Prafulla Mahindrakar can you chime in here or @David Espejo (he/him) or @Yee
r

Rio

05/23/2023, 2:24 PM
Look, maybe I'm just approaching this from the wrong angle, I'm not above admitting that. There's just nothing that seems to indicate that this shouldn't work.
k

Ketan (kumare3)

05/23/2023, 2:44 PM
We know that auth without TLS does not work, I cannot remember off the top of my head
p

Prafulla Mahindrakar

05/23/2023, 4:59 PM
@Rio can you share your flyteadmin config and flytectl config .
r

Rio

05/23/2023, 5:05 PM
Sorry @Prafulla Mahindrakar , I'm afk. What are you specifically looking for? For flytectl I've created am oauth2 client that can only do client credentials grants. And I've setup that config as such using 
ClientSecret
as a method. 
insecure: true
as mentioned before. Both the 
all
and 
offline
scopes are also set but are optional as they get included by default by keycloak. Flyteadmin has been configured with 
useAuth
, external auth server, user auth set to a separate oauth2 client that can only do pkce, and the third party set to the same flytectl client as previously mentioned.
p

Prafulla Mahindrakar

05/23/2023, 5:29 PM
Seems the scope claim is coming as 
"scope": "openid profile email all offline",
Whereas we expect it to be 
scp: …..
https://github.com/flyteorg/flyteadmin/blob/master/auth/authzserver/provider.go#L38 We can add a change to support this but most of the providers that the community has deployed have not run into this particular AFAIK but i might be wrong. I can provide a sandbox build of flyteadmin for you to try and see if that works for you and if this something configurable in keycloak to use scp to send the scope claims then that would work too
r

Rio

05/23/2023, 5:33 PM
No this seems to be used inside of claims, not scopes. This is the place that seems to return the error https://github.com/flyteorg/flyteadmin/blob/master/auth/interceptor.go#L21
p

Prafulla Mahindrakar

05/23/2023, 5:39 PM
heres the function which sets the scope in the identity context from the raw claims https://github.com/flyteorg/flyteadmin/blob/master/auth/authzserver/claims_verifier.go#L15
r

Rio

05/23/2023, 5:46 PM
Alright, so if i can get keycloak to add the scp claim to the token this should work? Did I miss any docs around this? Side question, am I the only one trying to run flyte in this way? Looking at this solution it would have failed any way as this requires specific configuration on the IDP side...
p

Prafulla Mahindrakar

05/23/2023, 5:50 PM
I think adding the scp claim should work. Haven’t encountered people using keycloak much or they might have noticed and solved it by themselves and not updated the community docs for it. Hopefully if you get this working then we can update the docs on them
Also they might not have used flytectl , which is only grpc client which has this check unfortunately
r

Rio

05/23/2023, 5:55 PM
It's EOD here so I'll check it out tomorrow. Am I right in understanding that flytectl is not used for automation? I'm fairly new to flyte but i was under the impression this was the go-to client to interact with the api...
p

Prafulla Mahindrakar

05/23/2023, 5:59 PM
yes it is used and we use okta for our auth provider and many folks have used Azure AD and auth0 where we didnt run into this. we used internally for all CI using flytectl and many other companies do aswell Also probably this authorization piece was missed on the http handler which is used by flyteconsole but is there for flytectl interaction. I am pretty sure this should cause issues even for you running workflows since propeller which also uses grpc would fail in communicating with admin .
r

Rio

05/23/2023, 6:02 PM
I'll need to do a bit more verification then. Any way thanks for the hints. Tomorrow I'll double check if internal components really don't have any issues and if adding tls to all components will fix it.
p

Prafulla Mahindrakar

05/23/2023, 6:04 PM
Sure. Let us know how it goes.
r

Rio

05/26/2023, 1:17 PM
Just a quick update. I did confirm that internal components were having the same auth issues as mentioned before. Notably the same error with regards to a missing required scope. Sadly I have not attempted to setup TLS for the internal components because of time constraints. Instead we opted for disabling flyte authentication and use the oauth2-proxy to allow user authentication, while relying on cluster level mTLS to secure communication between services.
I expect to revisit this in the future but I now simply don't have the time for it sadly.
Anyway, thanks for all of your input!
p

Prafulla Mahindrakar

05/30/2023, 9:48 PM
Sad that you had to use a workaround for it but good that you could make some progress on it. I see spring security which is widely used framework for java does support this also uses both formats when parsing out the scopes, which might be what needs to be done even in flyteadmin for supporting cases like yours https://github.com/spring-projects/spring-security/blob/main/oauth2/oauth2-resourc[…]ver/resource/authentication/JwtGrantedAuthoritiesConverter.java You can help with a contribution on this or as mentioned i can help create a sandbox build which you can try deploying in you environment and check
r

Rio

05/31/2023, 8:36 PM
Sadly my time on that project has passed. I could revisit it in the future if the contracts involve flyte again but I'm not going to put personal time into this particular project currently, I'm sorry. I am confused though as to why that scope is not documented. If I'd have implemented full on oauth2 + oidc authentication over TLS for both users and internal clients using the client credentials grant, they would have still failed because the IDP doesn't set the 
scp
claim and the docs don't mention you needing to configure the IDP to include that claim.
p

Prafulla Mahindrakar

05/31/2023, 9:35 PM
This is case when things are not conformant to the RFC and each OIDC provider implements them and have these pitfalls. In your case the provider you are using is conformant to use scope but the ones we have used like okta,azure ad, auth0 seems to use scp claims by default . there is no additional configuration that was needed to do this and hence probably was never included in the docs. This is always a challenge with multiple implementations of the RFC and each provider some time choosing to deviate.
r

Rio

05/31/2023, 9:52 PM
I understand. Not sure if your test suite uses the services you mentioned but it might be nice to include some popular open source IDPs like keycloak, dex or ory hydra. I find it interesting that I'm the first to run into this though...
p

Prafulla Mahindrakar

05/31/2023, 11:00 PM
Currently there are no integration tests for each provider and each OSS consumer runs the functional tests suite enabling auth on there own provider and report if they run into issues. Internally we use okta. Heres the link to repo containing the suites. Docs is the place where we have been collaborating with the community to find whats the difference across providers and the community is updating there findings.
4 Views
Powered by