https://flyte.org logo
Join Slack
Powered by
Question: does `flyte-binary` just not support TLS...
# flyte-deployment
g
Question: does 
flyte-binary
just not support TLS on the ingress resources? How is TLS currently being done if not using an Ingress?
j
it was designed primarily for AWS and GCP where the cert is set in an annotation. it would be a small change to add support for a TLS block to the ingress object. i believe someone else wanted to a while back, but i haven’t seen a PR for it
g
I got around it by manually creating and kustomizing Ingress objects; not ideal but works for now while I am experimenting.
j
can you paste the final ingress object (redacted). i’ll push a change to the chart if i find time
g
@jeev I only added the following to both the http and grpc ingresses:
Copy code
spec:

  # ...

  tls:
    - hosts:
        - <host>
      secretName: flyte-tls
g
@jeev saw you added this feature, thanks! I'm not sure how to use the list block though, do you have any examples?
also I'm guessing this hasn't actually been relased yet?
j
yes unreleased, but you can use the nightly chart
the tls list block is passed through to the ingress object as-is. i don’t have an example, but it should be pretty straightforward.
g
how do I reference the nightly chart? Will give this a shot and report back
j
Use “oci://ghcr.io/flyteorg/helm-charts/flyte-binary” as the chart path and “0.0-<COMMIT-SHA>” as the version.
g
so latest is 
0.0-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0
? right? I see neither 
ingressClassName
nor httpTls changes reflected when doing 
kubectl describe ingress flyte-flyte-binary-http
Copy code
ingress:
    host: <redacted>
    # TODO - update chart once tls support lands to add this
    grpcTls:
      - hosts: 
        - <redacted>
        secretName: cluster-wildcard-tls
    httpTls:
      - hosts: 
        - <redacted>
        secretName: cluster-wildcard-tls
    create: true
    ingressClassName: nginx
    commonAnnotations:
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "false"
      # TODO - once "ingressClassName" support lands, delete this
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    grpcAnnotations:
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
j
kubectl get ingress -o yaml
hmm maybe we should add a “tls” config so users can keep things dry, consistent with the other configs
are your grpcTls and httpTls configs the same @Gopal Vashishtha ?
g
Copy code
apiVersion: v1
items:
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>: '[{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/console","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/console/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/api","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/api/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/healthcheck","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/v1/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/.well-known","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/.well-known/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/login","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/login/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/logout","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/logout/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/callback","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/callback/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/me","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/config","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/config/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/oauth2","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/oauth2/*","allNodes":false}]'
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "false"
    creationTimestamp: "2023-08-08T15:01:48Z"
    generation: 1
    labels:
      <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
      <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
      <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
      <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.8.1
    name: flyte-flyte-binary-http
    namespace: flyte
    resourceVersion: "553670"
    uid: 42e2247e-7ded-42ba-b1d0-fb1c1dc1ad08
  spec:
    rules:
    - host: <http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>
      http:
        paths:
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /console
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /console/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /api
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /api/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /healthcheck
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /v1/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /.well-known
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /.well-known/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /login
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /login/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /logout
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /logout/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /callback
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /callback/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /me
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /config
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /config/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /oauth2
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /oauth2/*
          pathType: ImplementationSpecific
  status:
    loadBalancer:
      ingress:
      - ip: 10.48.3.19
      - ip: 10.48.3.23
      - ip: 52.235.149.87
      - ip: 52.244.67.12
-
yes they're the same
j
Copy code
@@ -202,6 +202,15 @@ ingress:
             name: use-annotation
       path: /
       pathType: Exact
+  ingressClassName: foobar
+  httpTls:
+    - hosts:
+        - foo.bar
+      secretName: foo
+  grpcTls:
+    - hosts:
+        - foo.bar
+      secretName: foo
 rbac:
   extraRules:
   - apiGroups:
produces:
Copy code
@@ -854,6 +854,7 @@ metadata:
     <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
     <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
 spec:
+  ingressClassName: "foobar"
   rules:
   - http:
       paths:
@@ -942,6 +943,10 @@ spec:
         path: /flyteidl.service.SignalService/*
         pathType: ImplementationSpecific
     host: "<redacted>"
+    tls:
+      - hosts:
+        - foo.bar
+        secretName: foo
and
Copy code
spec:
+  ingressClassName: "foobar"
   rules:
   - http:
       paths:
@@ -1110,3 +1116,7 @@ spec:
         path: /oauth2/*
         pathType: ImplementationSpecific
     host: "<redacted>"
+    tls:
+      - hosts:
+        - foo.bar
+        secretName: foo
for me.
and now i can just do:
Copy code
@@ -202,6 +202,11 @@ ingress:
             name: use-annotation
       path: /
       pathType: Exact
+  ingressClassName: foobar
+  tls:
+    - hosts:
+        - foo.bar
+      secretName: foo
 rbac:
   extraRules:
   - apiGroups:
are you sure you installed the correct helm chart @Gopal Vashishtha?
you can generate a manifest with this:
Copy code
helm template flyte-binary <oci://ghcr.io/flyteorg/helm-charts/flyte-binary> --namespace flyte --set deployment.image.tag=sha-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0 --version 0.0-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0 --values values.yaml > generated.yaml
and inspect it.
g
When I use the command you sent, the generated.yaml looks correct jeev. The problem is I'm actually running flyte as a dependency in a helm chart. When I do this:
Copy code
dependencies:
- name: flyte-binary
  version: "0.0-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0"
  repository: "<oci://ghcr.io/flyteorg/helm-charts/flyte-binary>"
I get the following error from 
helm dependency build
Copy code
Save error occurred:  could not download <oci://ghcr.io/flyteorg/helm-charts/flyte-binary/flyte-binary>: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to <https://ghcr.io/token?scope=repository%3Aflyteorg%2Fhelm-charts%2Fflyte-binary%2Fflyte-binary%3Apull&service=ghcr.io>: 403 Forbidden
I worked around the issue by doing 
helm pull
. I now see:
Copy code
W0809 06:50:42.034001   83117 warnings.go:70] unknown field "spec.rules[0].tls"
W0809 06:50:42.617311   83117 warnings.go:70] unknown field "spec.rules[0].tls"
looking at the docs, shouldn't "tls" be a sibling of rules, not a child?
j
ah looks like i made a mistake. will fix
can you take a look at that?
you can even try it out as a local chart.
g
Yeah this looks right
j
did you have a chance to try it out @Gopal Vashishtha?
g
@jeev I can't figure out how to reference your nightly build in my helm chart. Can you suggest syntax I can put in a Chart.yml to list the chart as a dependency?
see this error I was getting above
ah wait I got it
14 Views
Open in Slack
PreviousNext
Powered by