https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
g

Greg Linklater

07/26/2023, 12:13 PM
Question: does
flyte-binary
just not support TLS on the ingress resources? How is TLS currently being done if not using an Ingress?
j

jeev

07/26/2023, 1:10 PM
it was designed primarily for AWS and GCP where the cert is set in an annotation. it would be a small change to add support for a TLS block to the ingress object. i believe someone else wanted to a while back, but i haven’t seen a PR for it
g

Greg Linklater

07/26/2023, 6:33 PM
I got around it by manually creating and kustomizing Ingress objects; not ideal but works for now while I am experimenting.
j

jeev

07/27/2023, 6:08 AM
can you paste the final ingress object (redacted). i’ll push a change to the chart if i find time
g

Greg Linklater

07/27/2023, 9:34 AM
@jeev I only added the following to both the http and grpc ingresses:
Copy code
spec:

  # ...

  tls:
    - hosts:
        - <host>
      secretName: flyte-tls
g

Gopal Vashishtha

08/08/2023, 1:13 PM
@jeev saw you added this feature, thanks! I'm not sure how to use the list block though, do you have any examples?
also I'm guessing this hasn't actually been relased yet?
j

jeev

08/08/2023, 1:33 PM
yes unreleased, but you can use the nightly chart
the tls list block is passed through to the ingress object as-is. i don’t have an example, but it should be pretty straightforward.
g

Gopal Vashishtha

08/08/2023, 2:14 PM
how do I reference the nightly chart? Will give this a shot and report back
j

jeev

08/08/2023, 2:17 PM
Use “oci://ghcr.io/flyteorg/helm-charts/flyte-binary” as the chart path and “0.0-<COMMIT-SHA>” as the version.
g

Gopal Vashishtha

08/08/2023, 3:05 PM
so latest is
0.0-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0
? right? I see neither
ingressClassName
nor httpTls changes reflected when doing
kubectl describe ingress flyte-flyte-binary-http
Copy code
ingress:
    host: <redacted>
    # TODO - update chart once tls support lands to add this
    grpcTls:
      - hosts: 
        - <redacted>
        secretName: cluster-wildcard-tls
    httpTls:
      - hosts: 
        - <redacted>
        secretName: cluster-wildcard-tls
    create: true
    ingressClassName: nginx
    commonAnnotations:
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "false"
      # TODO - once "ingressClassName" support lands, delete this
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    grpcAnnotations:
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
j

jeev

08/08/2023, 3:07 PM
kubectl get ingress -o yaml
hmm maybe we should add a “tls” config so users can keep things dry, consistent with the other configs
are your grpcTls and httpTls configs the same @Gopal Vashishtha ?
g

Gopal Vashishtha

08/08/2023, 3:27 PM
Copy code
apiVersion: v1
items:
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>: '[{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/console","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/console/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/api","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/api/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/healthcheck","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/v1/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/.well-known","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/.well-known/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/login","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/login/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/logout","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/logout/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/callback","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/callback/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/me","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/config","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/config/*","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/oauth2","allNodes":false},{"addresses":["10.48.3.19","10.48.3.23","52.235.149.87","52.244.67.12"],"port":80,"protocol":"HTTP","serviceName":"flyte:flyte-flyte-binary-http","ingressName":"flyte:flyte-flyte-binary-http","hostname":"<http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>","path":"/oauth2/*","allNodes":false}]'
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "false"
    creationTimestamp: "2023-08-08T15:01:48Z"
    generation: 1
    labels:
      <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
      <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
      <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
      <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.8.1
    name: flyte-flyte-binary-http
    namespace: flyte
    resourceVersion: "553670"
    uid: 42e2247e-7ded-42ba-b1d0-fb1c1dc1ad08
  spec:
    rules:
    - host: <http://flyte.redacted.redacted.com|flyte.redacted.redacted.com>
      http:
        paths:
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /console
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /console/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /api
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /api/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /healthcheck
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /v1/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /.well-known
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /.well-known/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /login
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /login/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /logout
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /logout/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /callback
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /callback/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /me
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /config
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /config/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /oauth2
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyte-flyte-binary-http
              port:
                number: 8088
          path: /oauth2/*
          pathType: ImplementationSpecific
  status:
    loadBalancer:
      ingress:
      - ip: 10.48.3.19
      - ip: 10.48.3.23
      - ip: 52.235.149.87
      - ip: 52.244.67.12
-
yes they're the same
j

jeev

08/08/2023, 3:37 PM
Copy code
@@ -202,6 +202,15 @@ ingress:
             name: use-annotation
       path: /
       pathType: Exact
+  ingressClassName: foobar
+  httpTls:
+    - hosts:
+        - foo.bar
+      secretName: foo
+  grpcTls:
+    - hosts:
+        - foo.bar
+      secretName: foo
 rbac:
   extraRules:
   - apiGroups:
produces:
Copy code
@@ -854,6 +854,7 @@ metadata:
     <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
     <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
 spec:
+  ingressClassName: "foobar"
   rules:
   - http:
       paths:
@@ -942,6 +943,10 @@ spec:
         path: /flyteidl.service.SignalService/*
         pathType: ImplementationSpecific
     host: "<redacted>"
+    tls:
+      - hosts:
+        - foo.bar
+        secretName: foo
and
Copy code
spec:
+  ingressClassName: "foobar"
   rules:
   - http:
       paths:
@@ -1110,3 +1116,7 @@ spec:
         path: /oauth2/*
         pathType: ImplementationSpecific
     host: "<redacted>"
+    tls:
+      - hosts:
+        - foo.bar
+        secretName: foo
for me.
and now i can just do:
Copy code
@@ -202,6 +202,11 @@ ingress:
             name: use-annotation
       path: /
       pathType: Exact
+  ingressClassName: foobar
+  tls:
+    - hosts:
+        - foo.bar
+      secretName: foo
 rbac:
   extraRules:
   - apiGroups:
are you sure you installed the correct helm chart @Gopal Vashishtha?
you can generate a manifest with this:
Copy code
helm template flyte-binary <oci://ghcr.io/flyteorg/helm-charts/flyte-binary> --namespace flyte --set deployment.image.tag=sha-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0 --version 0.0-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0 --values values.yaml > generated.yaml
and inspect it.
g

Gopal Vashishtha

08/09/2023, 1:40 PM
When I use the command you sent, the generated.yaml looks correct jeev. The problem is I'm actually running flyte as a dependency in a helm chart. When I do this:
Copy code
dependencies:
- name: flyte-binary
  version: "0.0-9cbd3a2a0abc0a3978460bc0eb4eb1c3e01991e0"
  repository: "<oci://ghcr.io/flyteorg/helm-charts/flyte-binary>"
I get the following error from
helm dependency build
Copy code
Save error occurred:  could not download <oci://ghcr.io/flyteorg/helm-charts/flyte-binary/flyte-binary>: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to <https://ghcr.io/token?scope=repository%3Aflyteorg%2Fhelm-charts%2Fflyte-binary%2Fflyte-binary%3Apull&service=ghcr.io>: 403 Forbidden
I worked around the issue by doing
helm pull
. I now see:
Copy code
W0809 06:50:42.034001   83117 warnings.go:70] unknown field "spec.rules[0].tls"
W0809 06:50:42.617311   83117 warnings.go:70] unknown field "spec.rules[0].tls"
looking at the docs, shouldn't "tls" be a sibling of rules, not a child?
j

jeev

08/09/2023, 2:15 PM
ah looks like i made a mistake. will fix
can you take a look at that?
you can even try it out as a local chart.
g

Gopal Vashishtha

08/09/2023, 6:40 PM
Yeah this looks right
j

jeev

08/09/2023, 6:48 PM
did you have a chance to try it out @Gopal Vashishtha?
g

Gopal Vashishtha

08/10/2023, 10:48 PM
@jeev I can't figure out how to reference your nightly build in my helm chart. Can you suggest syntax I can put in a Chart.yml to list the chart as a dependency?
see this error I was getting above
ah wait I got it
11 Views