Chris Antenesse

    Chris Antenesse

    1 month ago
    howdy! i’ve deployed Flyte in AWS manually in EKS per the docs. things look good from a deployment perspective - the deployment, pods, services, and ingresses are up. i can hit the console web UI at the expected hostname. two things i havent done are get our SSL certs working on the ingress yet or set an authentication method. but i wanted to get the
    flytectl
    config set and be able to interact with the cluster via the command line with a minimum configuration before doing those two things. my minimal config looks like this
    admin:
     endpoint: dns:///admin.flyte.us3.predictap.com
     insecure: true
    my
    flytectl version
    output is this
    {
      "App": "flytectl",
      "Build": "62b86f6",
      "Version": "0.6.7",
      "BuildTime": "2022-08-11 11:33:31.781304 -0500 CDT m=+0.021446651"
    }%
    i think my config is good enough, i ran
    flytectl config validate
    chrisantenesse@Chriss-MacBook-Pro-2 ~ % flytectl config validate
    Couldn't find a config file.
    Validated config file successfully.
    but when i do something like
    flytectl get projects
    , i’m getting this
    {"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection closed","ts":"2022-08-11T11:35:12-05:00"}
    {"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-08-11T11:35:12-05:00"}
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-08-11T11:35:12-05:00"}
    Error: rpc error: code = Unavailable desc = connection closed
    {"json":{},"level":"error","msg":"rpc error: code = Unavailable desc = connection closed","ts":"2022-08-11T11:35:12-05:00"}
    i jumped on both flyteadmin pods shell and was able to confirm that traffic coming through the ingress was actually hitting the pods. basically ran
    watch -n1 netstat -anp
    and watched a connection get established. i also ran
    tcpdump
    locally and watched my local machine make the outbound request, etc. so i dont think this is due to a misconfiguration in the k8s side of things. i ran a
    kubectl logs …
    on both pods, but never saw entries from the flyteadmin server indicating that something happened (good or bad). im new to the flyte world and appreciate the help in advance!
    y

    Yuvraj

    1 month ago
    Are you sure, Your ingress and flyte deployment is in good shape, Can you provide me output of this command
    kubectl get ingress,pod -n flyte
    Chris Antenesse

    Chris Antenesse

    1 month ago
    figured this out on my end. summary - the ingress setup doesn’t work quite right.
    oops, sent at the same time 🙂
    y

    Yuvraj

    1 month ago
    can you give me output of
    kubectl get ingress,pod -n flyte
    ?
    Chris Antenesse

    Chris Antenesse

    1 month ago
    kubectl get ingress,pod -n flyte
    NAME                                        CLASS    HOSTS                           ADDRESS                                                                   PORTS   AGE
    <http://ingress.networking.k8s.io/flyte-core|ingress.networking.k8s.io/flyte-core>        <none>   <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>   <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>   80      26m
    <http://ingress.networking.k8s.io/flyte-core-grpc|ingress.networking.k8s.io/flyte-core-grpc>   <none>   <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>   <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>   80      26m
    
    NAME                                     READY   STATUS    RESTARTS   AGE
    pod/datacatalog-ff6d66cf5-vc8lt          1/1     Running   0          26m
    pod/datacatalog-ff6d66cf5-zwfq6          1/1     Running   0          26m
    pod/flyte-pod-webhook-595f7b6858-62fjn   1/1     Running   0          26m
    pod/flyteadmin-7f5d5ffc47-7ht64          1/1     Running   0          26m
    pod/flyteadmin-7f5d5ffc47-7w8t5          1/1     Running   0          26m
    pod/flyteconsole-565d6765b9-6vwln        1/1     Running   0          26m
    pod/flyteconsole-565d6765b9-kwdrv        1/1     Running   0          26m
    pod/flytepropeller-74bf956f6c-8x2f4      1/1     Running   0          26m
    pod/flytepropeller-74bf956f6c-qdczb      1/1     Running   0          26m
    pod/flytescheduler-65597b6ccf-8thg5      1/1     Running   0          26m
    pod/syncresources-5457d6d655-5vkjs       1/1     Running   0          26m
    y

    Yuvraj

    1 month ago
    Now can you give me output of
    kubectl get ingress -oyaml -n flyte
    Chris Antenesse

    Chris Antenesse

    1 month ago
    apiVersion: v1
    items:
    - apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
      kind: Ingress
      metadata:
        annotations:
          <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
          <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
          <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
          <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
        creationTimestamp: "2022-08-11T17:17:30Z"
        generation: 1
        labels:
          <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
        name: flyte-core
        namespace: flyte
        resourceVersion: "8414401"
        uid: dfda6080-dbde-434d-bb2d-a2e7657b7cc5
      spec:
        rules:
        - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
          http:
            paths:
            - backend:
                service:
                  name: ssl-redirect
                  port:
                    name: use-annotation
              path: /*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteconsole
                  port:
                    number: 80
              path: /console
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteconsole
                  port:
                    number: 80
              path: /console/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /api
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /api/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /healthcheck
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /v1/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /.well-known
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /.well-known/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /login
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /login/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /logout
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /logout/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /callback
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /callback/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /me
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /config
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /config/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /oauth2
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /oauth2/*
              pathType: ImplementationSpecific
      status:
        loadBalancer:
          ingress:
          - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
    - apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
      kind: Ingress
      metadata:
        annotations:
          <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
          <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
          <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
          <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
          <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
          <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
        creationTimestamp: "2022-08-11T17:17:30Z"
        generation: 3
        labels:
          <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
        name: flyte-core-grpc
        namespace: flyte
        resourceVersion: "8420898"
        uid: d222f010-bf87-4ddc-ac10-9b05ce59c9f0
      spec:
        rules:
        - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
          http:
            paths:
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AdminService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AdminService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.DataProxyService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.DataProxyService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AuthMetadataService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AuthMetadataService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.IdentityService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.IdentityService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /grpc.health.v1.Health
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /grpc.health.v1.Health/*
              pathType: ImplementationSpecific
      status:
        loadBalancer:
          ingress:
          - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
    y

    Yuvraj

    1 month ago
    in your ingress can you change the annotations to
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
    to
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP2
    Chris Antenesse

    Chris Antenesse

    1 month ago
    done
    trying again in 1m
    that didnt work
    i can provide the output of those command again
    apiVersion: v1
    items:
    - apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
      kind: Ingress
      metadata:
        annotations:
          <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
          <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
          <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
          <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
        creationTimestamp: "2022-08-11T17:17:30Z"
        generation: 1
        labels:
          <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
        name: flyte-core
        namespace: flyte
        resourceVersion: "8414401"
        uid: dfda6080-dbde-434d-bb2d-a2e7657b7cc5
      spec:
        rules:
        - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
          http:
            paths:
            - backend:
                service:
                  name: ssl-redirect
                  port:
                    name: use-annotation
              path: /*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteconsole
                  port:
                    number: 80
              path: /console
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteconsole
                  port:
                    number: 80
              path: /console/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /api
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /api/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /healthcheck
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /v1/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /.well-known
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /.well-known/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /login
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /login/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /logout
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /logout/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /callback
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /callback/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /me
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /config
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /config/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /oauth2
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 80
              path: /oauth2/*
              pathType: ImplementationSpecific
      status:
        loadBalancer:
          ingress:
          - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
    - apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
      kind: Ingress
      metadata:
        annotations:
          <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: HTTP2
          <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
          <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
          <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
          <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
          <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: HTTP2
        creationTimestamp: "2022-08-11T17:17:30Z"
        generation: 3
        labels:
          <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
        name: flyte-core-grpc
        namespace: flyte
        resourceVersion: "8421910"
        uid: d222f010-bf87-4ddc-ac10-9b05ce59c9f0
      spec:
        rules:
        - host: <http://admin.flyte.us3.predictap.com|admin.flyte.us3.predictap.com>
          http:
            paths:
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AdminService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AdminService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.DataProxyService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.DataProxyService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AuthMetadataService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.AuthMetadataService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.IdentityService
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /flyteidl.service.IdentityService/*
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /grpc.health.v1.Health
              pathType: ImplementationSpecific
            - backend:
                service:
                  name: flyteadmin
                  port:
                    number: 81
              path: /grpc.health.v1.Health/*
              pathType: ImplementationSpecific
      status:
        loadBalancer:
          ingress:
          - hostname: <http://a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com|a1fe6821b1e9c47eca9b9a18b2379e34-1655614568.us-east-1.elb.amazonaws.com>
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
    a

    allen

    1 month ago
    My 2c after I ran into the same issue see this thread • make sure to setup the SSL as per the instructions (before that I was getting similar flytectl errors) • make sure to use the values-eks file provided in the thread • ensure HTTP2 instead of GRPC is set, delete and recreate the ingresses to be sure