https://flyte.org logo
n

Nicholas LoFaso

02/14/2022, 7:37 PM
Hi @jeev @Sören Brunk @Nelson Arapé are any of you using Identity Aware Proxy (IAP) with GCP? cc: @Justin Tyberg
j

jeev

02/14/2022, 7:37 PM
we (freenome) are. everything is behind IAP
🙏 1
notably though we dont have it correctly plugged into flyte's auth system
j

Justin Tyberg

02/14/2022, 7:40 PM
Hi @jeev. I work with Nick. I’m curious about your set up. 1. Are you using Google external HTTPS load balancer as the ingress controller? Did you replace contour? 2. Do you have flyte auth disabled?
j

jeev

02/14/2022, 7:41 PM
we've basically disabled auth on flyte and depend on IAP to protect the frontend
👍 1
j

Justin Tyberg

02/14/2022, 7:41 PM
that’s what i figured
are you using google’s load balancer?
j

jeev

02/14/2022, 7:41 PM
yes we are
👍 1
j

Justin Tyberg

02/14/2022, 7:41 PM
or still contour?
j

jeev

02/14/2022, 7:41 PM
not contour
j

Justin Tyberg

02/14/2022, 7:42 PM
great. and did you have to configure BackendConfigs for each flyte service, with health checks, etc.?
and change the flyte services to
NodePort
?
i started down this path, • allow GCP to spin up an external load balancer for the flyte ingress • switched the flyte services to
NodePort
• load balancer health checks failed. my guess is that the load balancer could not infer a health check for pods with multiple containers
j

jeev

02/14/2022, 7:46 PM
we define backend services via terraform that are 1:1 with k8s services
and yes, nodeport for all "external" services
we did have to open up firewall to allow googleHC to hit our backends
does the k8s hc pass?
j

Justin Tyberg

02/14/2022, 7:49 PM
right. and did you need custom health checks for the backend configs? or was the load balancer able to figure it out?
well, it passed for 2/5 services
i hit the brakes after running into the health check issue. i need to spin this config up again. but it’s good to know that someone has got this working
j

jeev

02/14/2022, 7:52 PM
we do have a health checks defined in terraform. but this is kinda specific to how we expose our services
we had it working before with just gke ingress, which infers from the deployment
j

Justin Tyberg

02/14/2022, 7:53 PM
ok. maybe i don’t have enough fw ports open for all the flyte service ports
j

jeev

02/14/2022, 7:53 PM
if k8s hcs pass, but google's fail, it should be telling
j

Justin Tyberg

02/14/2022, 7:55 PM
yeah. let me keep going down this path. i was worried that if we put IAP in front of every flyte service, then somehow they would not be able to communicate.
n

Nicholas LoFaso

02/14/2022, 7:56 PM
Thanks Jeev! that’s great to hear you have it working, and I appreciate the insights. Helps to know we’re not going down the wrong path
j

jeev

02/14/2022, 7:58 PM
we use this now though: https://github.com/GoogleCloudPlatform/gke-autoneg-controller but we had it working before with just gke ingress
👀 1
i cant try to find the config
j

Justin Tyberg

02/14/2022, 8:09 PM
that would be awesome
k

Ketan (kumare3)

02/14/2022, 8:09 PM
cc @jeev / @Justin Tyberg would either of you please summarize and add this to github discussions - might help some folks
j

Justin Tyberg

02/14/2022, 8:10 PM
sure. once i get it working, i’d be happy to share
❤️ 1
j

jeev

02/14/2022, 8:11 PM
we def dont have a "reference" setup for GKE, but i think that would be super helpful. happy to help contribute to that.
j

Justin Tyberg

02/15/2022, 8:30 PM
so i’ve made some progress using GCP Identity Aware Proxy for authorization to flyte. for context… • we’re trying to spin up multiple GKE clusters running flyte, • we don’t want to configure OAuth for each flyte cluster (configure domains and redirect URIs, etc). • we’d rather use GCP load balancer (in favor of contour) and push TLS termination and auth to the load balancer. the idea is just let GCP and IAP do its thing. my set up • no contour component • GCP external HTTPS load balancer, created by GKE using the flyte ingress • flyteadmin and flyteconsole services point to backend configs that enable IAP • flyte auth disabled,
common.adminServer.server.security.useAuth: false
i was able to get the browser authentication working. IAP authenticates me because i’m in the google org. i can get to the flyte console.
however,
flytectl
is not working.
Copy code
flytectl get projects

{"json":{"src":"client.go:193"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-02-15T15:24:31-05:00"}
{"json":{"src":"client.go:66"},"level":"info","msg":"Initialized Admin client","ts":"2022-02-15T15:24:31-05:00"}
Error: rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: received the unexpected content-type "text/html; charset=UTF-8"
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: received the unexpected content-type \"text/html; charset=UTF-8\"","ts":"2022-02-15T15:24:31-05:00"}
i’m looking at the flyte auth appendix now
to see what the load balancer is trying to hit
15 Views