https://flyte.org logo
h

Hampus Rosvall

05/09/2022, 5:46 PM
Hey, as an extra layer of security I want to authenticate users in the ALB by adding oidc Ingress annotations. How would I configure `flytectl`to still be able to authenticate users when the OIDC flow happens in the ALB rather than in the Flyte Console? On another note, is it possible to deploy Flyte internally and still be able to authenticate users/deploy workflows in CI/CD? I guess the topics are quite different, but it would be interesting to see what choice you prefer and how that is implemented in practice. Thanks 🙂
k

Ketan (kumare3)

05/09/2022, 11:31 PM
i thought flytectl should still work if you are using standard pkce flow? cc @Haytham Abuelfutuh
i dont understand the second question
h

Hampus Rosvall

05/10/2022, 5:13 AM
I am not able to get it to work, using
authType: Pkce
and
insecure: false
I am getting following error when trying to communicate with the backend. The 302 response is the HTTP response from login page to IdP, as the oidc flow is happening in the ALB rather than in the application itself. For the second question I was referring to this security advisory where the suggested workaround is disabling flyte console availability on the internet. Not sure how that would work in practice, i.e., how do you do communicate with the Flyte Admin either locally or from a CI/CD workflow?
Copy code
~: flytectl get projects
Error: rpc error: code = Unknown desc = Found: HTTP status code 302; transport: received the unexpected content-type "text/html"
k

Ketan (kumare3)

05/10/2022, 1:15 PM
@Hampus Rosvall you are no longer vulnerable to that
We have yanked all older versions
h

Hampus Rosvall

05/10/2022, 1:51 PM
Yes, that’s awesome! But for security reasons it would be really nice adding an additional security layer in the ALB. Have you done something like that previously?
k

Ketan (kumare3)

05/10/2022, 3:21 PM
I think someone else should answer that / my expertise is missing here
h

Hampus Rosvall

05/10/2022, 3:31 PM
Right, no problem! Is there anyone that would know we can loop in?
k

Ketan (kumare3)

05/10/2022, 3:32 PM
Ya @Haytham Abuelfutuh or @Prafulla Mahindrakar or @Yee or @Sören Brunk
❤️ 1
p

Prafulla Mahindrakar

05/11/2022, 7:25 AM
Hi @Hampus Rosvall why not use the same domian name that you use for flyteconsole in flytectl config.yaml and have the ingress handle the redirection for you . Also the ssl termination happens at the elb which has ingress rules for both flyteconsole which uses REST endpoints and for flytectl which uses GRPC . Flyteadmin doesn’t itself come up with ssl and relies on ssl termination to happen in the upper layers. If you have tried this and it fails please check the ingress is getting applied correctly . There should be two ingress in flyte name space one for grpc and other for non-grpc For your second question , you can look at how we run our functional tests using flytectl https://github.com/flyteorg/flytetools/blob/d4d9c38d99cf8ba7648783bf417448ffe19d683d/functional-tests/config.yaml#L7 Here it relies on the default authType of client_secrets https://github.com/flyteorg/flyteidl/blob/654427ef4027c6b48d997abe8770c2e8b915e84a/clients/go/admin/config.go#L32 (default 0)
h

Hampus Rosvall

05/12/2022, 8:13 AM
Hi Prafulla, I have tried to modify the domain name for the flyteconsole in the
~/.flyte/config.yaml
but I am getting the following error below as the are using the oidc Ingress annotations i.e., the endpoint hits the
Copy code
~: flytectl get projects
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-05-12T10:11:43+02:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-05-12T10:11:43+02:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-05-12T10:11:43+02:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-05-12T10:11:43+02:00"}
{"json":{"src":"client.go:183"},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unknown desc = Found: HTTP status code 302; transport: received the unexpected content-type \"text/html\"","ts":"2022-05-12T10:11:44+02:00"}
{"json":{"src":"client.go:188"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-05-12T10:11:44+02:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-05-12T10:11:44+02:00"}
Error: rpc error: code = Unknown desc = Found: HTTP status code 302; transport: received the unexpected content-type "text/html"
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unknown desc = Found: HTTP status code 302; transport: received the unexpected content-type \"text/html\"","ts":"2022-05-12T10:11:44+02:00"}
As we are using oidc ALB annotations i.e.,
<http://alb.ingress.kubernetes.io/auth-type|alb.ingress.kubernetes.io/auth-type>: oidc
and pointing to our IDP.
p

Prafulla Mahindrakar

05/12/2022, 9:37 AM
Flyte does support OIDC based authentication and OAUTH 2 server through flyteadmin directly https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html And any reason you are not using this. Using the ALB sounds interesting for doing OIDC and offloading this from flyteadmin. Given that case are you bringing up flyteadmin with auth i.e is useAuth property set to true in config. Given an offloaded process of doing auth, then admin shouldn’t be enabled for this. We haven’t tried such a setup but would be interesting to see what all it takes to make it work but first would recommend using the flyteadmin auth route to get started. With flyteadmin based auth, you automatically get audit info at flyte entity level, eg : who created the execution, With the ALB based OIDC i am not sure how this user info gets available to admin . Since we store in auth context such info
h

Hampus Rosvall

05/12/2022, 1:16 PM
Hi Prafulla, thanks for quick response! The main reason for doing the OIDC authentication in the ALB is security, we have used FlyteAdmin for the auth flow previously but an extra layer of security/off loading auth to the ALB would be really nice. The useAuth property is thus false currently
p

Prafulla Mahindrakar

05/13/2022, 6:23 AM
Hi @Hampus Rosvall you might want to check if the annotation(
<http://alb.ingress.kubernetes.io/auth-type|alb.ingress.kubernetes.io/auth-type>: oidc)
you added exists on both the ingress objects. for grpc helm installer creates a new ingress object
kubectl get ingress -n flyte
h

Hampus Rosvall

05/13/2022, 7:47 AM
Yes, both
flyte-core
and
flyte-core-grpc
has the same annotations
p

Prafulla Mahindrakar

05/13/2022, 10:30 AM
I think admin is forcing the auth here, We should probably add authType None which is missing in admin auth config for cases like yours
I guess that would just remove the auth warning but wont solve the issue completely. Seems after initializing the admin client the request is getting a 302. I am assuming you can port forward and have flytectl talk to admin and fetch the data correctly , Its only when going through the alb with the annotation you are facing this issue
h

Hampus Rosvall

05/13/2022, 12:25 PM
Yes, I am using port-forwarding currently. But was curious to see if we could move away from that and do some special configuration point
flytectl
to our endpoint and still be able to handle authentication. But perhaps there is no way to do so?
p

Prafulla Mahindrakar

05/13/2022, 12:50 PM
None atm. But we should figure this out . Do you mind opening an issue for this .
h

Hampus Rosvall

05/18/2022, 6:53 AM
Totally missed your reply. Yes of course I can do that 🙂
đź‘Ť 1
43 Views