https://flyte.org
Join Slack
Menu
Home
Archive
<https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-identity-aware-proxy#configuring-...
# flyte-on-gcp
h
https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-identity-aware-proxy#configuring-your-flyte-deployment-to-use-iap is there a way to configure flyte with ssl so that a Istio service mesh is not required for IaP?
c
You need TLS (can be a self signed cert) between the GCP load balancer and the backend when using http2/grpc. In this case this affects flyteadmin.
In the guide above I used istio to configure TLS between the load balancer and the istio ingress gateway which is the first backend behind the ingress. This way, flyteadmin doesn’t have to be aware of TLS.
You might be able to do the same with an nginx between the ingress and flyteadmin.
Flyteadmin itself in theory also can use a TLS cert but in practice I found it doesn’t work as it prevents e.g. flytepropeller from talking to it.
All of the above is only required if you want a GCP Ingress with IaP.
If you just want SSL, I recommend you use the standard nginx config from the helm chart.
Happy to help you through this config if you have a requirement to deploy with iap 🙂
h
yeah i wonder if flytepropeller -> ssl -> admin is just a bug or configuration issue though. I imagine if using a self signed cert, a custom Root CA might be needed on the propeller side
you would need to mount the CA certificate to the propeller instances, and use something like 
sudo update-ca-certificates
or similar
i plan on testing this soon
c
I think it’s a bug
There is an option in the admin client used by propeller which configures it to just accept any certificate admin presents. But some connections to admin don’t honor this config.
See here
h
Copy code
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
yeah i think this is the issue. I believe if you create a custom CA and add it as a trusted CA, then this might work
👍 1
c
The istio/cert manager setup is nice because it auto-renews the cert.
But it would be nice to not have to use istio
h
yeah. I think for my use case though, i will need to use a service mesh eventually, and possibly i would go Anthos route. This is because we have gpu workstations on prem. Anthos is based on istio and would allow for running GKE on prem
👍 1
c
If you try to go the istio route and have trouble with it, I’d be happy to jump on a quick call and help debug at least 🤷
h
thanks. ill let you know. I'm going to deploy with just gcp auth for now. I will also try testing the CA cert route. If i can get that working ill put together a write up on it
c
Here this “insecureSkipVerify” I mentioned is used.
And the CA file is loaded here.
Problem is that this is only for the grpc client.
I think the connection that actually fails is not grpc but when trying to get a token from the token endpoint.
h
hmmmm ok
c
Here the http client is instantiated that is used to e.g. retrieve the token
It doesn’t load any CA cert file or have the insecure skip verify option.
My gut feeling is that this is where it would need to be added.
h
the http client should use the trusted CA certificates from the operating system though
c
That I never tried 👍
h
its an extra step in the helm chart to mount the cert and run that update ca command
c
That is a good catch, that might have helped me back when trying to get this to work 👍
But you will need something that regenerates the cert and restarts flyteadmin and propeller when the cert expires.
With istio+cert manager this works automatically.
But I think it would be good for users to have both options.
So I’d be curious whether you can get it to work.
h
I'm fairly confident i can get it to work, the question is if i should just use istio so i don't have to mange a bunch of self signed certs... I am also wondering if the better path is to use Anthos for the service mesh instead which would be a managed service.
c
Haven’t used Anthos myself yet.
Do you still get to create istio VirtualService and Gateway objects?
h
not sure, i haven't looked deeply into either Anthos or istio. What i do know is that Anthos is require to run GKE on prem, and we have workloads we want to run that only run on specific hardware that we have on prem. I might have problems with drivers, but that is a future me problem
👍 1
c
With istio I could help, anthos only if it’s close to istio
h
https://cloud.google.com/service-mesh/docs/overview "Anthos Service Mesh is powered by Istio, a highly configurable and powerful open source service mesh platform, with tools and features that enable industry best practices."
https://forfuncsake.github.io/post/2017/08/trust-extra-ca-cert-in-go-app/ also this might be an option, this would require code changes to add the ca certs to the http clients
👍 1
c
Deploy a 
Gateway
and 
VirtualService
for the frontend service
This is literally just istio
This are just istio objects.
So my flyte + istio + IaP guide should just work out of the box on an anthos cluster.
👍 1
gratitude thank you 1
If you need anthos anyways for the on-prem machine, I’d go for that.
28 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by