https://flyte.org logo
Join Slack
Powered by
Hi. First time trying to deploy Flyte on AWS follo...
# flyte-deployment
l
Hi. First time trying to deploy Flyte on AWS following these instructions https://docs.flyte.org/en/latest/deployment/aws/manual.html After installing the Helm chart and verifying that the pods are running, I fail to get an address for my hosts:
Copy code
$ kubectl get ingress -n flyte
NAME              CLASS    HOSTS   ADDRESS   PORTS   AGE
flyte-core        <none>   *                 80      26m
flyte-core-grpc   <none>   *                 80      26m
Describing the ingress shows this
Copy code
kubectl describe ingress -n flyte
Name:             flyte-core
Namespace:        flyte
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /*               ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
              /console         flyteconsole:80 (192.168.103.57:8080,192.168.128.109:8080)
              /console/*       flyteconsole:80 (192.168.103.57:8080,192.168.128.109:8080)
              /api             flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /api/*           flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /healthcheck     flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /v1/*            flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /.well-known     flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /.well-known/*   flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /login           flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /login/*         flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /logout          flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /logout/*        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /callback        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /callback/*      flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /me              flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /config          flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /config/*        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /oauth2          flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /oauth2/*        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-west-2:582526512915:certificate/6c75c8f4-04a1-4aa7-81fa-59c7241e52ba
              <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
              <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: [{"HTTP": 80}, {"HTTPS":443}]
              <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
              <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
              <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
              <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
              <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Events:
  Type     Reason             Age                  From     Message
  ----     ------             ----                 ----     -------
  Warning  FailedDeployModel  5m1s (x18 over 26m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.


Name:             flyte-core-grpc
Namespace:        flyte
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /flyteidl.service.AdminService            flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.AdminService/*          flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.DataProxyService        flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.DataProxyService/*      flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.AuthMetadataService     flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.AuthMetadataService/*   flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.IdentityService         flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.IdentityService/*       flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /grpc.health.v1.Health                    flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /grpc.health.v1.Health/*                  flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
              <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-west-2:582526512915:certificate/6c75c8f4-04a1-4aa7-81fa-59c7241e52ba
              <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
              <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: [{"HTTP": 80}, {"HTTPS":443}]
              <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
              <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
              <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
              <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
              <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
              <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
Events:
  Type     Reason             Age                  From     Message
  ----     ------             ----                 ----     -------
  Warning  FailedDeployModel  5m4s (x19 over 27m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.
Any ideas how to best debug?
y
those instructions are a bit dated unf. we’ll look to clean them up soon hopefully.
they shouldn’t have annotations for both nginx and alb.
what ingress controller are you using? alb i assume since that’s what the instructions have?
l
yes, using ALB
is there anything I can change in 
values-eks.yaml
as a workaround?
y
sorry was afk
looking a bit more now at their docs
are there any logs from the ingress controller itself?
l
how can I access those logs? I’m a bit of a k8s noob
sorry, that was an easy thing to do 😅
Copy code
{
  "level": "error",
  "ts": 1671213485.8348856,
  "logger": "controller.ingress",
  "msg": "Reconciler error",
  "name": "flyte",
  "namespace": "",
  "error": "InvalidParameter: 1 validation error(s) found.\n- minimum field value of 1, CreateTargetGroupInput.Port.\n"
}
this was the only error log entry
y
can you get -o yaml the ingress also?
both of them, flyte-core and the grpc one
this here might be relevant
subnet needs to be tagged with 
<http://kubernetes.io/role/elb=1|kubernetes.io/role/elb=1>
sorry we’ve been using nginx as the ingress controller for a spell now, been a while since i’ve played around with alb
we honestly need to delete our instructions and just point users to the official docs (esp since we have no way of knowing or keeping up with the updates)
can you also 
k -n flyte get service -o wide
l
do you mean pass 
-o yaml
to the 
k describe ingress
call?
Copy code
kubectl get service -n flyte -o wide
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                    AGE     SELECTOR
datacatalog         NodePort    10.100.184.184   <none>        8089:32690/TCP,88:31839/TCP,89:32201/TCP   3h12m   <http://app.kubernetes.io/instance=flyte,app.kubernetes.io/name=datacatalog|app.kubernetes.io/instance=flyte,app.kubernetes.io/name=datacatalog>
flyte-pod-webhook   ClusterIP   10.100.226.35    <none>        443/TCP                                    3h12m   app=flyte-pod-webhook
flyteadmin          ClusterIP   10.100.200.84    <none>        80/TCP,81/TCP,87/TCP,10254/TCP             3h12m   <http://app.kubernetes.io/instance=flyte,app.kubernetes.io/name=flyteadmin|app.kubernetes.io/instance=flyte,app.kubernetes.io/name=flyteadmin>
flyteconsole        ClusterIP   10.100.211.80    <none>        80/TCP                                     3h12m   <http://app.kubernetes.io/instance=flyte,app.kubernetes.io/name=flyteconsole|app.kubernetes.io/instance=flyte,app.kubernetes.io/name=flyteconsole>
thanks so much for the help 🙏
y
k -n flyte get ingress -o yaml
yeah i don’t think the datacatalog service should be a nodeport.
l
Copy code
apiVersion: v1
items:
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
        { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
      <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-west-2:*************:certificate/6c75c8f4-04a1-4aa7-81fa-59c7241e52ba
      <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
      <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
      <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
      <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
    creationTimestamp: "2022-12-16T17:58:03Z"
    finalizers:
    - group.ingress.k8s.aws/flyte
    generation: 1
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    name: flyte-core
    namespace: flyte
    resourceVersion: "408320"
    uid: 3d84d59a-08fe-41d9-bea3-305413ddd483
  spec:
    rules:
    - http:
        paths:
        - backend:
            service:
              name: ssl-redirect
              port:
                name: use-annotation
          path: /*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteconsole
              port:
                number: 80
          path: /console
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteconsole
              port:
                number: 80
          path: /console/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /api
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /api/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /healthcheck
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /v1/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /.well-known
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /.well-known/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /login
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /login/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /logout
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /logout/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /callback
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /callback/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /me
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /config
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /config/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /oauth2
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /oauth2/*
          pathType: ImplementationSpecific
  status:
    loadBalancer: {}
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>: '{"Type": "redirect", "RedirectConfig":
        { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
      <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
      <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-west-2:*************:certificate/6c75c8f4-04a1-4aa7-81fa-59c7241e52ba
      <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
      <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
      <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
      <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
    creationTimestamp: "2022-12-16T17:58:03Z"
    finalizers:
    - group.ingress.k8s.aws/flyte
    generation: 1
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    name: flyte-core-grpc
    namespace: flyte
    resourceVersion: "408218"
    uid: 114b0b7e-ede6-43f1-8baa-7871600b81c5
  spec:
    rules:
    - http:
        paths:
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AdminService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AdminService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.DataProxyService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.DataProxyService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.IdentityService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.IdentityService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /grpc.health.v1.Health
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /grpc.health.v1.Health/*
          pathType: ImplementationSpecific
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
curious, has anyone attempted an ECS deployment of Flyte? Could be a good option for AWS folks who are not as comfortable with k8s - and potentially much more amenable to Cloudformation/Terraform style deployment
Ok, confirmed that the Github issue fix did the trick for me
adding
Copy code
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
to the annotations created the ALB correctly
y
sorry working on another issue right now… let me come back to this in a bit
l
I did not have to tag my subnets, FWIW
y
oh nice
good to know
l
now to work on authentication
thanks @Yee and hope this thread is useful for anyone encountering the same problem in the future
k
Great to see you in the channel @Lawrence Lee thank you for trying out Flyte flyte
f
I followed the same deployment guide today and had to make the same adjustment, i.e. adding
Copy code
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
to the 
eks-values.yaml
. That guide is really lovely and much more helpful than the deployment paths on the latest website. I also found that the guide was - apart from that flag above - in great shape still. Any thoughts on adding it to the current version of the website?
k
thank you for the comment @Ferdinand von den Eichen. the old guide we felt was overly verbose for most people.
that is why we made a simpler one
d
cc @Niels Bantilan
n
the old guide we felt was overly verbose for most people.
yeah, on the other hand it sounds like it helped people get setup… based on @Ferdinand von den Eichen’s feedback it may be worth adding at least a link to those manual setup docs as a stop-gap. For context, Ferdinand, maintaining those guides manually was very challenging… however, if we do put up a link to those legacy guides, would you be able to help us figure out what’s out-dated? We are working on a reference implementation for AWS and GCP via terraform, which will nullify the need for the manual docs setup and will simplify how you can stand up a Flyte cluster on the cloud.
f
I totally understand the effort 😱 at the same time, the high level (current) guide was good for getting a basic understanding of all pieces involved, but lacks many of the crucial details to ensure a smooth first setup. I could imagine that Flyte is about to gain a lot of traction, so first setup experience should be smooth sailing on the major clouds. The Issue you linked sounds incredibly promising in that regard though, so I’ll leave it up to you guys if you think it will cover the new user path sufficiently. It sounds like it does actually! Regarding the (outdated) article: The only change I needed to make was adding
Copy code
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
to the 
eks-values.yaml
. Everything else was still on point.
k
@Ferdinand von den Eichen honestly if you can help maintain we can keep the article as under the hood?
f
I / we at Kineo don’t mind contributing back at all, that’s the magic of OS, right? 🫶 That being said, I’m not sure how easy it is to contribute here, since it is more than “just” a PR against README.md. Can you point me in the right direction?
k
Cc @David Espejo (he/him) please comment
But we are a very open community and love contributions
d
We do! @Ferdinand von den Eichen Here some instructions to contribute to docs: https://docs.flyte.org/projects/flytekit/en/latest/contributing.html#contribute-to-documentation If any question let us know!
f
I started a PR here: https://github.com/flyteorg/flyte/pull/3343 After checking the docs again I realised, that the change is actually not needed in the docs, but rather in the eks-values.yaml itself. I opened a PR, any help is appreciated. In particular with making sure relevant tests run (if there are any for the EKS deployment?)
k
@David Espejo (he/him) can you help merge this please
a
@Ferdinand von den Eichen I’m a bit confused. This comment on the above github issue said to add that annotation to the service (so putting it under 
flyteadmin.service.annotations
and 
flyteconsole.service.annotations
. But apparently it’ll just work as an annotation on the ingress?
d
@Alex Papanicolaou the 
<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>
annotation can be applied both at the Service and Ingress level, with a higher precedence for the Service level. In this case, the annotation for Ingress will be applied unless there's another at the Service level but I guess due to the Flyte architecture and how Ingress is the main entrance point from flytekit, it makes sense to just annotate the Ingress resource
a
Thanks @David Espejo (he/him)! That makes sense and makes the ingress setup simple.
191 Views
Open in Slack
PreviousNext
Powered by