https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
s

Shihgian Lee

06/02/2022, 7:20 PM
Hi All, we are deploying flyte to our kubernetes cluster. We are at the SSL Certificate step. We don’t want to expose flyte to outside of our organization. But, it looks like the SSL Certificate for GKE is only for external HTTPS. Are there other options we can use so that we don’t expose flyte to outside world? Appreciate any helps from the community.
@Ketan (kumare3) Sorry to bother you. This is currently blocking our kubernetes deployment. I was wondering if you or someone on your team can give us some advice on this. Thanks in advance!
k

Ketan (kumare3)

06/02/2022, 10:11 PM
hi @Shihgian Lee let me think
you can use LetsEncrypt?
s

Shihgian Lee

06/02/2022, 10:13 PM
@Ketan (kumare3) Thanks so much for your reply! I am not familiar with LetsEncrypt. I am working with our devops. We can look into it and then get back to you?
k

Ketan (kumare3)

06/02/2022, 10:14 PM
yes
s

Shihgian Lee

06/02/2022, 10:51 PM
Hi @Ketan (kumare3), thank you for taking the time to chat with us. Here is the recap of the flyte production on GKE questions we have for your team: 1. What configuration file should we be looking for to change ingress from
external
to
internal
? 2. After we have successfully changed the ingress to
internal
, we can’t use the Google Managed Cert anymore. What cert option do we have for internal ingress? Thank you in advance for your help!
k

Ketan (kumare3)

06/03/2022, 3:53 AM
cc @Prafulla Mahindrakar / @Pradithya Aria Pura/ @jeev / @Yuvraj does any of you know the answer to this. They want to enable FlyteAdmin to be within their VPN
j

jeev

06/03/2022, 3:59 AM
we use an external load balancer with a google cert, but put everything behind IAP so that our users have to authenticate with google before being granted permissions to view the sites. this causes issues with talking to flyteadmin programmatically from outside our cluster. we just port-forward for that. our default dev envs which also run on GKE don't have this problem. i don't have experience running behind a VPN on GKE unfortunately, sorry.
🙏 1
p

Prafulla Mahindrakar

06/03/2022, 5:50 AM
We haven’t deployed such a setup but these instructions might help you @Shihgian Lee https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balance-ingress And adding the gce-internal annotations for the ingress https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-gcp.yaml and
Copy code
<http://cloud.google.com/neg|cloud.google.com/neg>: '{"ingress": true}'
on the service account for flyteadmin
🙏 1
s

Shihgian Lee

06/03/2022, 10:42 PM
Thanks for the resources, @Prafulla Mahindrakar I added the annotation to `flyteadmin -> service`:
Copy code
service:
    annotations:
      <http://cloud.google.com/neg|cloud.google.com/neg>: '{"ingress": true}'
      # Required for the ingress to properly route grpc traffic to grpc port
      <http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'
I did try to add it to
serviceAccount
annotation. But, it didn’t seem right based on the documentation. So, I added it to
service
instead. Under the
common->ingress
, I added and commented out a few entries based on the documentation you shared:
Copy code
common:
  ingress:
    host: "{{ .Values.userSettings.hostName }}"
    tls:
      enabled: true
    annotations:
      # <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      # <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
      # <http://cert-manager.io/issuer|cert-manager.io/issuer>: "letsencrypt-production"
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: "gce-internal"
      <http://kubernetes.io/ingress.allow-http|kubernetes.io/ingress.allow-http>: "false"
      <http://kubernetes.io/ingress.global-static-ip-name|kubernetes.io/ingress.global-static-ip-name>: dev-porch-com-example
    # --- separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx.
    # separateGrpcIngress: true
    # --- Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled.
    # separateGrpcIngressAnnotations:
      # <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
I managed to execute helm successfully. However, my ingress looks like the following:
Copy code
kubectl -n flyte get ingress
NAME         CLASS    HOSTS                ADDRESS   PORTS     AGE
flyte-core   <none>   <http://flyte.porchpoc.com|flyte.porchpoc.com>             80, 443   45s
The
<https://flyte.porchpoc.com/console>
cannot be reached. I think it is because the
Address
is missing. There seems to be some errors for the ingress:
Copy code
kubectl get events --all-namespaces --field-selector involvedObject.kind=Ingress
NAMESPACE   LAST SEEN   TYPE      REASON   OBJECT               MESSAGE
flyte       44m         Normal    Sync     ingress/flyte-core   Scheduled for sync
flyte       43m         Warning   Sync     ingress/flyte-core   Error syncing to GCP: error running load balancer syncing routine: error initializing translator env: secrets "flyte-flyte-tls" not found
flyte       5m4s        Normal    Sync     ingress/flyte-core   Scheduled for sync
flyte       2m8s        Warning   Sync     ingress/flyte-core   Error syncing to GCP: error running load balancer syncing routine: error initializing translator env: secrets "flyte-flyte-tls" not found
Any ideas what I can try next? Thanks!
p

Prafulla Mahindrakar

06/06/2022, 11:18 AM
Hi @Shihgian Lee, can you try turning off the tls secret in your values file https://github.com/flyteorg/flyte/blob/b28acd09b407acc2ac99535c0a16a40a051585a1/charts/flyte-core/values-gcp.yaml#L149 We have it off in other deployments too
190 Views