https://flyte.org
Join Slack
Hi <@U017K8AJBAN> <@U0165SDP3BQ> <@UP4PLM3F0> are any of you using Identity Aware Proxy (<IAP>) with...
# flyte-deployment
r
Hi @freezing-boots-56761 @boundless-pizza-95864 @better-air-29727 are any of you using Identity Aware Proxy (IAP) with GCP? cc: @gifted-raincoat-59712
f
we (freenome) are. everything is behind IAP
🙏 1
notably though we dont have it correctly plugged into flyte's auth system
g
Hi @freezing-boots-56761. I work with Nick. I’m curious about your set up. 1. Are you using Google external HTTPS load balancer as the ingress controller? Did you replace contour? 2. Do you have flyte auth disabled?
f
we've basically disabled auth on flyte and depend on IAP to protect the frontend
👍 1
g
that’s what i figured
are you using google’s load balancer?
f
yes we are
👍 1
g
or still contour?
f
not contour
g
great. and did you have to configure BackendConfigs for each flyte service, with health checks, etc.?
and change the flyte services to 
NodePort
?
i started down this path, • allow GCP to spin up an external load balancer for the flyte ingress • switched the flyte services to 
NodePort
• load balancer health checks failed. my guess is that the load balancer could not infer a health check for pods with multiple containers
f
we define backend services via terraform that are 1:1 with k8s services
and yes, nodeport for all "external" services
we did have to open up firewall to allow googleHC to hit our backends
does the k8s hc pass?
g
right. and did you need custom health checks for the backend configs? or was the load balancer able to figure it out?
well, it passed for 2/5 services
i hit the brakes after running into the health check issue. i need to spin this config up again. but it’s good to know that someone has got this working
f
we do have a health checks defined in terraform. but this is kinda specific to how we expose our services
we had it working before with just gke ingress, which infers from the deployment
g
ok. maybe i don’t have enough fw ports open for all the flyte service ports
f
if k8s hcs pass, but google's fail, it should be telling
g
yeah. let me keep going down this path. i was worried that if we put IAP in front of every flyte service, then somehow they would not be able to communicate.
f
r
Thanks Jeev! that’s great to hear you have it working, and I appreciate the insights. Helps to know we’re not going down the wrong path
f
we use this now though: https://github.com/GoogleCloudPlatform/gke-autoneg-controller but we had it working before with just gke ingress
👀 1
i cant try to find the config
g
that would be awesome
f
cc @freezing-boots-56761 / @gifted-raincoat-59712 would either of you please summarize and add this to github discussions - might help some folks
g
sure. once i get it working, i’d be happy to share
❤️ 1
f
we def dont have a "reference" setup for GKE, but i think that would be super helpful. happy to help contribute to that.
g
so i’ve made some progress using GCP Identity Aware Proxy for authorization to flyte. for context… • we’re trying to spin up multiple GKE clusters running flyte, • we don’t want to configure OAuth for each flyte cluster (configure domains and redirect URIs, etc). • we’d rather use GCP load balancer (in favor of contour) and push TLS termination and auth to the load balancer. the idea is just let GCP and IAP do its thing. my set up • no contour component • GCP external HTTPS load balancer, created by GKE using the flyte ingress • flyteadmin and flyteconsole services point to backend configs that enable IAP • flyte auth disabled, 
common.adminServer.server.security.useAuth: false
i was able to get the browser authentication working. IAP authenticates me because i’m in the google org. i can get to the flyte console.
however, 
flytectl
is not working.
Copy code
flytectl get projects

{"json":{"src":"client.go:193"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-02-15T15:24:31-05:00"}
{"json":{"src":"client.go:66"},"level":"info","msg":"Initialized Admin client","ts":"2022-02-15T15:24:31-05:00"}
Error: rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: received the unexpected content-type "text/html; charset=UTF-8"
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: received the unexpected content-type \"text/html; charset=UTF-8\"","ts":"2022-02-15T15:24:31-05:00"}
i’m looking at the flyte auth appendix now
to see what the load balancer is trying to hit
200 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by