https://flyte.org logo
Join Slack
Powered by
howdy! i’ve deployed Flyte in AWS manually in EKS ...
# flyte-support
c
howdy! i’ve deployed Flyte in AWS manually in EKS per the docs. things look good from a deployment perspective - the deployment, pods, services, and ingresses are up. i can hit the console web UI at the expected hostname. two things i havent done are get our SSL certs working on the ingress yet or set an authentication method. but i wanted to get the 
flytectl
config set and be able to interact with the cluster via the command line with a minimum configuration before doing those two things. my minimal config looks like this
Copy code
admin:
 endpoint: dns:///admin.flyte.us3.predictap.com
 insecure: true
my 
flytectl version
output is this
Copy code
{
  "App": "flytectl",
  "Build": "62b86f6",
  "Version": "0.6.7",
  "BuildTime": "2022-08-11 11:33:31.781304 -0500 CDT m=+0.021446651"
}%
i think my config is good enough, i ran 
flytectl config validate
Copy code
chrisantenesse@Chriss-MacBook-Pro-2 ~ % flytectl config validate
Couldn't find a config file.
Validated config file successfully.
but when i do something like 
flytectl get projects
, i’m getting this
Copy code
{"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection closed","ts":"2022-08-11T11:35:12-05:00"}
{"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-08-11T11:35:12-05:00"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-08-11T11:35:12-05:00"}
Error: rpc error: code = Unavailable desc = connection closed
{"json":{},"level":"error","msg":"rpc error: code = Unavailable desc = connection closed","ts":"2022-08-11T11:35:12-05:00"}
i jumped on both flyteadmin pods shell and was able to confirm that traffic coming through the ingress was actually hitting the pods. basically ran 
watch -n1 netstat -anp
and watched a connection get established. i also ran 
tcpdump
locally and watched my local machine make the outbound request, etc. so i dont think this is due to a misconfiguration in the k8s side of things. i ran a 
kubectl logs …
on both pods, but never saw entries from the flyteadmin server indicating that something happened (good or bad). im new to the flyte world and appreciate the help in advance!
g
Are you sure, Your ingress and flyte deployment is in good shape, Can you provide me output of this command
Copy code
kubectl get ingress,pod -n flyte
c
figured this out on my end. summary - the ingress setup doesn’t work quite right.
oops, sent at the same time 🙂
g
can you give me output of 
kubectl get ingress,pod -n flyte
?
c
Copy code
kubectl get ingress,pod -n flyte
NAME                                        CLASS    HOSTS                           ADDRESS                                                                   PORTS   AGE
<http://ingress.networking.k8s.io/flyte-core|ingress.networking.k8s.io/flyte-core>        <none>   <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>   <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>   80      26m
<http://ingress.networking.k8s.io/flyte-core-grpc|ingress.networking.k8s.io/flyte-core-grpc>   <none>   <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>   <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>   80      26m

NAME                                     READY   STATUS    RESTARTS   AGE
pod/datacatalog-ff6d66cf5-vc8lt          1/1     Running   0          26m
pod/datacatalog-ff6d66cf5-zwfq6          1/1     Running   0          26m
pod/flyte-pod-webhook-595f7b6858-62fjn   1/1     Running   0          26m
pod/flyteadmin-7f5d5ffc47-7ht64          1/1     Running   0          26m
pod/flyteadmin-7f5d5ffc47-7w8t5          1/1     Running   0          26m
pod/flyteconsole-565d6765b9-6vwln        1/1     Running   0          26m
pod/flyteconsole-565d6765b9-kwdrv        1/1     Running   0          26m
pod/flytepropeller-74bf956f6c-8x2f4      1/1     Running   0          26m
pod/flytepropeller-74bf956f6c-qdczb      1/1     Running   0          26m
pod/flytescheduler-65597b6ccf-8thg5      1/1     Running   0          26m
pod/syncresources-5457d6d655-5vkjs       1/1     Running   0          26m
g
Now can you give me output of 
kubectl get ingress -oyaml -n flyte
c
Copy code
apiVersion: v1
items:
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
    creationTimestamp: "2022-08-11T17:17:30Z"
    generation: 1
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    name: flyte-core
    namespace: flyte
    resourceVersion: "8414401"
    uid: dfda6080-dbde-434d-bb2d-a2e7657b7cc5
  spec:
    rules:
    - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
      http:
        paths:
        - backend:
            service:
              name: ssl-redirect
              port:
                name: use-annotation
          path: /*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteconsole
              port:
                number: 80
          path: /console
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteconsole
              port:
                number: 80
          path: /console/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /api
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /api/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /healthcheck
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /v1/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /.well-known
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /.well-known/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /login
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /login/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /logout
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /logout/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /callback
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /callback/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /me
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /config
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /config/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /oauth2
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /oauth2/*
          pathType: ImplementationSpecific
  status:
    loadBalancer:
      ingress:
      - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
    creationTimestamp: "2022-08-11T17:17:30Z"
    generation: 3
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    name: flyte-core-grpc
    namespace: flyte
    resourceVersion: "8420898"
    uid: d222f010-bf87-4ddc-ac10-9b05ce59c9f0
  spec:
    rules:
    - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
      http:
        paths:
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AdminService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AdminService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.DataProxyService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.DataProxyService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.IdentityService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.IdentityService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /grpc.health.v1.Health
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /grpc.health.v1.Health/*
          pathType: ImplementationSpecific
  status:
    loadBalancer:
      ingress:
      - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
g
in your ingress can you change the annotations to 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
to 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP2
c
done
trying again in 1m
that didnt work
i can provide the output of those command again
Copy code
apiVersion: v1
items:
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
    creationTimestamp: "2022-08-11T17:17:30Z"
    generation: 1
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    name: flyte-core
    namespace: flyte
    resourceVersion: "8414401"
    uid: dfda6080-dbde-434d-bb2d-a2e7657b7cc5
  spec:
    rules:
    - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
      http:
        paths:
        - backend:
            service:
              name: ssl-redirect
              port:
                name: use-annotation
          path: /*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteconsole
              port:
                number: 80
          path: /console
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteconsole
              port:
                number: 80
          path: /console/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /api
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /api/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /healthcheck
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /v1/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /.well-known
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /.well-known/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /login
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /login/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /logout
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /logout/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /callback
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /callback/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /me
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /config
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /config/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /oauth2
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 80
          path: /oauth2/*
          pathType: ImplementationSpecific
  status:
    loadBalancer:
      ingress:
      - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
      <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP2
    creationTimestamp: "2022-08-11T17:17:30Z"
    generation: 3
    labels:
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    name: flyte-core-grpc
    namespace: flyte
    resourceVersion: "8421910"
    uid: d222f010-bf87-4ddc-ac10-9b05ce59c9f0
  spec:
    rules:
    - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
      http:
        paths:
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AdminService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AdminService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.DataProxyService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.DataProxyService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.IdentityService
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.IdentityService/*
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /grpc.health.v1.Health
          pathType: ImplementationSpecific
        - backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /grpc.health.v1.Health/*
          pathType: ImplementationSpecific
  status:
    loadBalancer:
      ingress:
      - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
g
My 2c after I ran into the same issue see this thread • make sure to setup the SSL as per the instructions (before that I was getting similar flytectl errors) • make sure to use the values-eks file provided in the thread • ensure HTTP2 instead of GRPC is set, delete and recreate the ingresses to be sure
c
Hey all, I'm wondering if you have any more guidance how to set up flyte with nginx and get flytectl working I currently am getting 400s when I try access the grpc endpoint with flytectl
Copy code
flytectl register files --force --project flytesnacks --domain development --archive flyte-package.tgz --version 1
results in the following at the ingress
Copy code
99.153.68.49 - - [06/Oct/2022:01:31:01 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - eec91dddcd78fe41fb8702e5392cb06c
99.153.68.49 - - [06/Oct/2022:01:31:02 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - ab7ae5a0bf016f903df955dd87fd1928
99.153.68.49 - - [06/Oct/2022:01:31:02 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - c2a233d50e7eeb6964b7e488bd9d9bd8
99.153.68.49 - - [06/Oct/2022:01:31:02 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 355b47d9585ff0e89c80f2eeca9233ac
99.153.68.49 - - [06/Oct/2022:01:31:02 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - d87e5edbb620f7bf82d2a945b27179d3
99.153.68.49 - - [06/Oct/2022:01:31:02 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - c7a11b9533a865382c7022bd14335263
99.153.68.49 - - [06/Oct/2022:01:31:02 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 764f6f65d294fee42c5dc14445fe7a92
I am running the separate ingress and have tried both 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP2
and 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
but both give the above 400s
Seems like this might be an issue with nginx not supporting grpc without tls? https://github.com/kubernetes/ingress-nginx/issues/6736 Is there a way to register a workflow without grpc (http2)? For a couple reasons, we are locked into to nginx and in some scenarios without tls.
169 Views
Open in Slack
PreviousNext
Powered by