https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
k

Ketan (kumare3)

02/16/2022, 6:37 PM
@Nicholas LoFaso / @jeev / @Sören Brunk / @Pradithya Aria Pura / @Guillaume Perchais - and other GCP users. Nick has been trying to get IAP working with Flyte and here is an excerpt - Google documentation recommended we use an Identity Aware Proxy (IAP) with the Google HTTPS load balancer instead of contour. That approach almost worked, but there is an issue with the flyteadmin grpc healthcheck and HTTPS load balancer. — Would any of you folks be interested in joining a working group / meeting to help with this problem. Nick has graciously accepted to help document the findings and this will make Flyte even more reliable and accessible on GCP. cc @Prafulla Mahindrakar cc @Aijamal (do you think someone from GCP can help with this too?)
👍 2
Ok I will schedule one for next week?
or maybe Friday? cc @Nicholas LoFaso
👍 2
j

Justin Tyberg

02/16/2022, 7:25 PM
Summary Requirements • We need to authorize users of Flyte services on all GKE clusters. Only MSAT employees should be able to access these services. • We will provision multiple GKE clusters, each with their own Flyte installation. • We want the provisioning and bootstrapping of Flyte clusters to be fully automated. • Ideally, we can leverage Google IAP to protect all deployments at the Google Organization and Project levels. • Ideally, we can push TLS termination and user authorization to the edge components (e.g., using GKE Ingress for HTTPS load balancing). Flyte comes with auth out-of-the-box • Flyte has authentication built-in, once you enable it. • Currently, we configure Flyte to use OAuth client authorization. This has one major drawback: we have to manually configure the OAuth domains and redirect URIs for each Flyte deployment. • We cannot automate creating OAuth clients. ◦ If you use the Google APIs to create new OAuth clients, then you cannot use the GCP console to update the client (to add domains and redirects, see here). ◦ Google lacks APIs to configure IAP origins and redirect URIs programmatically. What we’ve tried • GCP external HTTPS load balancer as the ingress controller for the Flyte ingress • not using contour, the default ingress controller Flyte uses • Flyte authentication disabled (
common.adminServer.server.security.useAuth: false
) • custom backend-configs for the Flyte GKE services, with GCP IAP enabled What IS working • OK: custom backend configs and Google health checks from external HTTPS load balancer to Flyte services running on HTTP port 80 • OK: IAP authorization to Flyte HTTPS workloads through the browser What IS NOT working • FAIL: Google backend configs and health checks to gRPC service endpoint. ◦ GCP: use only TCP for grpc health checks: “For backend services that use the gRPC protocol, use only gRPC or TCP health checks. Do not use HTTP(S) or HTTP/2 health checks.” ◦ GKE ingress: you can only use HTTP, HTTPS, or HTTP/2 for health checks: “PROTOCOL used by probe systems for health checking. The 
BackendConfig
 only supports creating health checks using the HTTP, HTTPS, or HTTP2 protocols.” • FAIL:
flytectl
CLI connecting to
flyteadmin
behind GCP external HTTPS load balancer
a

austin

02/16/2022, 7:49 PM
Happy to join WG. And, to help develop the solution ( if I am able to set aside the time / prioritize this ).
k

Ketan (kumare3)

02/16/2022, 7:49 PM
@austin thank you, let me send you an invite
a

austin

02/16/2022, 7:59 PM
“Only MSAT employees should be able to access these services.” <- is this a more specific requirement for an individual use case ( MSAT refers to a company )?
n

Nicholas LoFaso

02/16/2022, 8:05 PM
MSAT is short for MethaneSAT the company Justin and I work for. The requirement could be generalized to only registered users should be able to access these services
👍 1
k

Ketan (kumare3)

02/16/2022, 9:32 PM
cc @Haytham Abuelfutuh
@Justin Tyberg I had a few questions about some of the requirements, and why this decision for e.g.,
Copy code
We will provision multiple GKE clusters, each with their own Flyte installation.
You can use the same Flyte control plane with multiple GKE clusters and allow each project / domain etc to be routed to a specific cluster
is this not desirable?
j

Justin Tyberg

02/16/2022, 9:36 PM
• separate environments (e.g., dev, stage, prod) • separate business units/orgs
do many flyte users deploy flyte control plane on one cluster and use other clusters as “workers”?
i guess we are viewing the cluster w/ flyte as the deployable unit
k

Ketan (kumare3)

02/16/2022, 10:34 PM
@Justin Tyberg yes, we know both scenarios. The advantage of having one central Flyte is sharability. But again, this is not required, just interested in learning
j

Justin Tyberg

02/16/2022, 10:40 PM
i suppose its an option. maybe have a GKE cluster just for flyte. but each of our worker clusters could be 1000 nodes. something to consider.
k

Ketan (kumare3)

02/16/2022, 11:15 PM
ohh wow, i guess it depends on size of the workflow
but, ya start with separated
and if you need we can merge them together, but this cool
p

Pradithya Aria Pura

02/17/2022, 1:56 AM
@Arief Rahmansyah Can you join the WG, this is relevant for us
👍 1
k

Ketan (kumare3)

02/17/2022, 2:10 AM
We are meeting on Friday 9:00 am PT, but will schedule at a better time in the future
a

Arief Rahmansyah

02/17/2022, 2:46 AM
Yes, I would like to join the WG. However, I’m not sure I can join the meeting since it’s at 00:00 midnight at my timezone (GMT+7). Would be great if there’s a recording or meeting notes. Anyway, I’ll share my email to you so you can invite me to the WG/meeting @Ketan (kumare3)
k

Ketan (kumare3)

02/17/2022, 4:17 AM
Yes will do meeting notes
And then I think you guys should decide the time
a

austin

02/18/2022, 5:31 PM
@Ketan (kumare3) - hadnt’ seen a meeting/destination…. ( is that today/now, or next week )?
g

Guillaume Perchais

02/21/2022, 7:36 PM
I was off, have you guys been able to progress on this?
k

Ketan (kumare3)

02/21/2022, 8:03 PM
@Guillaume Perchais yes, I am going to send an email soon to GCP. There seems to be a problem in GCP load balancers and their support for grPC
101 Views