@Nicholas LoFaso / @jeev / @Sören Brunk / @Pradithya Aria Pura / @Guillaume Perchais - and other GCP users. Nick has been trying to get IAP working with Flyte and here is an excerpt - Google documentation recommended we use an Identity Aware Proxy (IAP) with the Google HTTPS load balancer instead of contour. That approach almost worked, but there is an issue with the flyteadmin grpc healthcheck and HTTPS load balancer. — Would any of you folks be interested in joining a working group / meeting to help with this problem. Nick has graciously accepted to help document the findings and this will make Flyte even more reliable and accessible on GCP. cc @Prafulla Mahindrakar cc @Aijamal (do you think someone from GCP can help with this too?)

Summary Requirements • We need to authorize users of Flyte services on all GKE clusters. Only MSAT employees should be able to access these services. • We will provision multiple GKE clusters, each with their own Flyte installation. • We want the provisioning and bootstrapping of Flyte clusters to be fully automated. • Ideally, we can leverage Google IAP to protect all deployments at the Google Organization and Project levels. • Ideally, we can push TLS termination and user authorization to the edge components (e.g., using GKE Ingress for HTTPS load balancing). Flyte comes with auth out-of-the-box • Flyte has authentication built-in, once you enable it. • Currently, we configure Flyte to use OAuth client authorization. This has one major drawback: we have to manually configure the OAuth domains and redirect URIs for each Flyte deployment. • We cannot automate creating OAuth clients. ◦ If you use the Google APIs to create new OAuth clients, then you cannot use the GCP console to update the client (to add domains and redirects, see here). ◦ Google lacks APIs to configure IAP origins and redirect URIs programmatically. What we’ve tried • GCP external HTTPS load balancer as the ingress controller for the Flyte ingress • not using contour, the default ingress controller Flyte uses • Flyte authentication disabled (

common.adminServer.server.security.useAuth: false

) • custom backend-configs for the Flyte GKE services, with GCP IAP enabled What IS working • OK: custom backend configs and Google health checks from external HTTPS load balancer to Flyte services running on HTTP port 80 • OK: IAP authorization to Flyte HTTPS workloads through the browser What IS NOT working • FAIL: Google backend configs and health checks to gRPC service endpoint. ◦ GCP: use only TCP for grpc health checks: “For backend services that use the gRPC protocol, use only gRPC or TCP health checks. Do not use HTTP(S) or HTTP/2 health checks.” ◦ GKE ingress: you can only use HTTP, HTTPS, or HTTP/2 for health checks: “PROTOCOL used by probe systems for health checking. The 

BackendConfig

 only supports creating health checks using the HTTP, HTTPS, or HTTP2 protocols.” • FAIL:

flytectl

CLI connecting to

flyteadmin

behind GCP external HTTPS load balancer

Happy to join WG. And, to help develop the solution ( if I am able to set aside the time / prioritize this ).

@austin thank you, let me send you an invite

“Only MSAT employees should be able to access these services.” <- is this a more specific requirement for an individual use case ( MSAT refers to a company )?

MSAT is short for MethaneSAT the company Justin and I work for. The requirement could be generalized to only registered users should be able to access these services

cc @Haytham Abuelfutuh

• separate environments (e.g., dev, stage, prod) • separate business units/orgs

@Justin Tyberg yes, we know both scenarios. The advantage of having one central Flyte is sharability. But again, this is not required, just interested in learning

i suppose its an option. maybe have a GKE cluster just for flyte. but each of our worker clusters could be 1000 nodes. something to consider.

ohh wow, i guess it depends on size of the workflow

@Arief Rahmansyah Can you join the WG, this is relevant for us

We are meeting on Friday 9:00 am PT, but will schedule at a better time in the future

Yes, I would like to join the WG. However, I’m not sure I can join the meeting since it’s at 00:00 midnight at my timezone (GMT+7). Would be great if there’s a recording or meeting notes. Anyway, I’ll share my email to you so you can invite me to the WG/meeting @Ketan (kumare3)

Yes will do meeting notes

@Ketan (kumare3) - hadnt’ seen a meeting/destination…. ( is that today/now, or next week )?

I was off, have you guys been able to progress on this?

@Guillaume Perchais yes, I am going to send an email soon to GCP. There seems to be a problem in GCP load balancers and their support for grPC