I have deployed flyte on k8s using the flyte-binar...
# flyte-deployment
n
I have deployed flyte on k8s using the flyte-binary helm chart. I set create=true for ingress in values.yaml and it created the dns records and 2 ingresses, one for http and one for grpc.
Copy code
NAME                        CLASS    HOSTS                                       ADDRESS                                                                            PORTS   AGE
orchestrator-grpc   <none>   <http://orchestrator.playground.cloud.abc.com|orchestrator.playground.cloud.abc.com>   <http://b37bbe12aafaf42a7b06f611f91b07bd-7362ba7e14b8b74e.elb.eu-central-1.amazonaws.com|b37bbe12aafaf42a7b06f611f91b07bd-7362ba7e14b8b74e.elb.eu-central-1.amazonaws.com>   80      3h41m
orchestrator-http   <none>   <http://orchestrator.playground.cloud.abc.com|orchestrator.playground.cloud.abc.com>   <http://b37bbe12aafaf42a7b06f611f91b07bd-7362ba7e14b8b74e.elb.eu-central-1.amazonaws.com|b37bbe12aafaf42a7b06f611f91b07bd-7362ba7e14b8b74e.elb.eu-central-1.amazonaws.com>   80      3h41m
1. When I list the ingresses, I get this. As you can see the hostnames for both the ingresses are same. Is this expected? I am told that this could be a problem. Also, is there anything needed to be done for the webhook service? like setting up an ingress or anything to handle https traffic? 2. We are using nginx ingress controller and cert-manager to provision certificates in k8s. To set up nginx to use the certificate provided by cert-manager, 2 things are required - a) annotation specifying the cert-manager / issuer -
<http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: "abc-issuer"
and b) in the ingress spec, a tls section needs to be added like so -
Copy code
tls:
  - hosts:
    - <http://echo1.example.com|echo1.example.com>
    - <http://echo2.example.com|echo2.example.com>
    secretName: echo-tls
As shown here in the cert-manager docs. However, the flyte-binary ingress template does not have these elements to be overridden from values.yaml. I tried adding the spec section in values.yaml, but this does not work.
Copy code
spec:
    tls:
      hosts:
        - "${app_name}.${env}.<http://cloud.abc.com|cloud.abc.com>"
      secretName: "${app_name}.${env}.<http://cloud.abc.com|cloud.abc.com>"
At this point, with the dns configured, the UI works, but without setting up SSL, grpc will not work, so we are not able to register workflow. Please let me know if anyone has faced this issue and found a resolution. Thanks!
d
HI @Nandakumar Raghu 1. Yes, it's expected. If you
describe
each one of the Ingress resources you'll see that, despite using the same host name, the controller will route traffic to the corresponding service depending on the path (multiplexor pattern). I don't think there's anything you need to do for the
webhook
service
2. In
values.yaml
under
commonAnnotations
you can add the required annotations for, in this case, Ingress resource:
Copy code
commonAnnotations:
  ingress:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: nameOfClusterIssuer
...
regarding the `tls`section, I'm looking at what can be done besides patching the deployed Ingress
definitely would be a great addition to Flyte the Hard Way (currently uses only ACM/Route53)
n
@David Espejo (he/him) Yes please! That will be helpful. There is a root commonAnnotations and there is an ingress.commonAnnotations. I have added the
<http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: nameOfClusterIssuer
in the latter and it did not work. That is why I am thinking it might need the
tls
section because it said so in the cert-manager docs. Should it be added in the root commonAnnotations as you have shown above?
d
I don't think so. Whatever you put as annotation there will be merged with the base config as an annotation, not a new
spec
section
n
Hi @David Espejo (he/him) / @jeev, any ideas on how to inject the tls section?
j
The cleanest way will be to extend the helm chart with support for specifying the tls block
@Nandakumar Raghu spent a bit more time reading through the thread. are you terminating SSL at the nginx ingress controller instead of ELB?
if terminating at ELB, you shouldn’t need cert-manager or the TLS block on the ingress object.
Also, for my own clarity, is the nginx ingress controller just for Flyte, or are there other applications served through the same controller?
n
We are terminating SSL at nginx ingress controller and the controller serves other applications as well.
j
Ok then we should add a TLS block to the helm chart ingress template. Would you be open to creating a PR with the change? :)
n
Sure 🙂 I just did that on our side to get it working!
r
@Nandakumar Raghu @jeev This is a great discussion and I am running into the same issue. Would either of you be able to point me to where this is being worked on or where there is a working version? Also, happy to contribute but still learning Flyte's project structure. 😅
j
@Raphael Guzman i haven’t seen a PR for a TLS block in the ingress object. @Nandakumar Raghu?
114 Views