https://flyte.org logo
Powered by
t

Tan Li

06/30/2022, 7:17 AM
Hi community, I was trying to setup flyte-sandbox locally on my k3s cluster. I followed this instruction https://docs.flyte.org/en/latest/deployment/sandbox.html#deployment . First I was stuck at the point that 
svclb-flyte-deps-contour-envoy-c519e84f-lnpvm
keeps pending in the namespace kube-system. I worked around this by upgradeing the flyte-deps config. However, I am still confronting the envoy pod pending issue. Terminal output are in the reply thread. Anyone know how should I fix this? In general, it seems to be some port allocation issue, but I really don’t have any idea how to debug this futher, lol. Really appreciate it! 🙏
👍 2
Here is the output from the kubectl get pods:
Copy code
$ kubectl get pods -n kube-system
NAME                                                   READY   STATUS    RESTARTS   AGE
local-path-provisioner-6c79684f77-2t6g4                1/1     Running   0          77m
coredns-d76bd69b-cnh8f                                 1/1     Running   0          77m
svclb-traefik-d46b5b65-pck27                           2/2     Running   0          77m
traefik-df4ff85d6-pn7xs                                1/1     Running   0          77m
metrics-server-7cd5fcb6b7-f4c8q                        1/1     Running   0          77m
svclb-minio-3bbacdee-whhs6                             2/2     Running   0          16m
svclb-flyte-deps-kubernetes-dashboard-e2d186b7-xxq8x   1/1     Running   0          16m
svclb-flyte-deps-contour-envoy-c519e84f-lnpvm          2/2     Running   0          12m

$ kubectl get pods -n flyte
NAME                                              READY   STATUS    RESTARTS   AGE
flyte-deps-contour-envoy-k8rmw                    0/2     Pending   0          18m
postgres-df78c4cfc-8q6sz                          1/1     Running   0          18m
flyte-deps-kubernetes-dashboard-8b7d858b7-78tgn   1/1     Running   0          18m
minio-67cb9c6dd7-rhppm                            1/1     Running   0          18m
flyte-deps-contour-contour-cd4d956d9-kkl2w        1/1     Running   0          18m
flyteconsole-845f5c44b-tmpsr                      1/1     Running   0          10m
syncresources-766786b498-rv5jx                    1/1     Running   0          10m
flytepropeller-8645d94f64-b9pzd                   1/1     Running   0          10m
datacatalog-9c879977b-chrmw                       1/1     Running   0          10m
flyte-pod-webhook-5bd8f7fcb8-f9tgm                1/1     Running   0          10m
flyteadmin-55d8886bd6-cvmf4                       1/1     Running   0          10m
I then inspect the envoy pod in namespace flyte by doing kubectl describe, the here is the output
Copy code
Name:           flyte-deps-contour-envoy-k8rmw
Namespace:      flyte
Priority:       0
Node:           <none>
Labels:         <http://app.kubernetes.io/component=envoy|app.kubernetes.io/component=envoy>
                <http://app.kubernetes.io/instance=flyte-deps|app.kubernetes.io/instance=flyte-deps>
                <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
                <http://app.kubernetes.io/name=contour|app.kubernetes.io/name=contour>
                controller-revision-hash=67bdb7bd55
                <http://helm.sh/chart=contour-7.10.1|helm.sh/chart=contour-7.10.1>
                pod-template-generation=1
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  DaemonSet/flyte-deps-contour-envoy
Init Containers:
  envoy-initconfig:
    Image:      <http://docker.io/bitnami/contour:1.20.1-debian-10-r53|docker.io/bitnami/contour:1.20.1-debian-10-r53>
    Port:       <none>
    Host Port:  <none>
    Command:
      contour
    Args:
      bootstrap
      /config/envoy.json
      --xds-address=flyte-deps-contour
      --xds-port=8001
      --resources-dir=/config/resources
      --envoy-cafile=/certs/ca.crt
      --envoy-cert-file=/certs/tls.crt
      --envoy-key-file=/certs/tls.key
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     10m
      memory:  50Mi
    Environment:
      CONTOUR_NAMESPACE:  flyte (v1:metadata.namespace)
    Mounts:
      /admin from envoy-admin (rw)
      /certs from envoycert (ro)
      /config from envoy-config (rw)
Containers:
  shutdown-manager:
    Image:      <http://docker.io/bitnami/contour:1.20.1-debian-10-r53|docker.io/bitnami/contour:1.20.1-debian-10-r53>
    Port:       <none>
    Host Port:  <none>
    Command:
      contour
    Args:
      envoy
      shutdown-manager
    Liveness:     http-get http://:8090/healthz delay=120s timeout=5s period=20s #success=1 #failure=6
    Environment:  <none>
    Mounts:
      /admin from envoy-admin (rw)
  envoy:
    Image:       <http://docker.io/bitnami/envoy:1.21.1-debian-10-r55|docker.io/bitnami/envoy:1.21.1-debian-10-r55>
    Ports:       8080/TCP, 8443/TCP, 8002/TCP
    Host Ports:  80/TCP, 443/TCP, 0/TCP
    Command:
      envoy
    Args:
      -c
      /config/envoy.json
      --service-cluster $(CONTOUR_NAMESPACE)
      --service-node $(ENVOY_POD_NAME)
      --log-level info
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:      10m
      memory:   50Mi
    Liveness:   http-get http://:8002/ready delay=120s timeout=5s period=20s #success=1 #failure=6
    Readiness:  http-get http://:8002/ready delay=10s timeout=1s period=3s #success=1 #failure=3
    Environment:
      CONTOUR_NAMESPACE:  flyte (v1:metadata.namespace)
      ENVOY_POD_NAME:     flyte-deps-contour-envoy-k8rmw (v1:metadata.name)
    Mounts:
      /admin from envoy-admin (rw)
      /certs from envoycert (rw)
      /config from envoy-config (rw)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  envoy-admin:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  envoy-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  envoycert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  envoycert
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule> op=Exists
                 <http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule> op=Exists
                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists
                 <http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule> op=Exists
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists
                 <http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule> op=Exists
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  5m41s                  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
  Warning  FailedScheduling  5m36s (x1 over 5m39s)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
I am curious, why the enovy requesting host’s 80, 443 and even 0 port? In my understanding based on reading the config on the github, this host is referring to the kube node right, not the actual host machine?
Copy code
envoy:
    Image:       <http://docker.io/bitnami/envoy:1.21.1-debian-10-r55|docker.io/bitnami/envoy:1.21.1-debian-10-r55>
    Ports:       8080/TCP, 8443/TCP, 8002/TCP
    Host Ports:  80/TCP, 443/TCP, 0/TCP
    Command:
      envoy
k

Kevin Su

06/30/2022, 8:26 AM
I just follow this instruction to deploy flyte on kind cluster, it works for me. it seems like one of the `Host Port`’s is already blocked by another pod. Did you install anything that isn’t flyte on your k3s cluster?
t

Tan Li

06/30/2022, 8:37 AM
Seems traefik is causing a lot of problem here. I am new to K3s actually, is there anyway to turn it off?
k

Kevin Su

06/30/2022, 9:28 AM
btw, you can try flytectl sandbox to use flyte. it will launch a docker that has k3s and flyte deployment inside it.
k

Ketan (kumare3)

06/30/2022, 1:57 PM
Thank you Kevin
t

Tan Li

07/04/2022, 2:13 AM
Hi @Kevin Su I tried to deploy again following the tutorial still no luck. The previous .sh file you showed me was deploying using docker, but I was following the kubernete’s version, i.e. I am trying to deploy it to a remote kubernetes cluster, not with local docker desktop. This time I spin up an entirely fresh new cluster, once the K3S cluster is up & running (nothing running on it besides default stuffs out of k3s box), I literally copy paste the following commands from the doc, but still no luck.
Copy code
helm repo add flyteorg <https://helm.flyte.org>

helm install -n flyte flyte-deps flyteorg/flyte-deps --create-namespace -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-deps/values-sandbox.yaml>

helm install flyte flyteorg/flyte-core -n flyte -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-sandbox.yaml> --wait
When I call 
kubectl get pods -n flyte
, the envoy proxy is still pending.
Copy code
NAME                                              READY   STATUS    RESTARTS   AGE
flyte-deps-contour-envoy-qwp6d                    0/2     Pending   0          7m9s
flyte-deps-contour-contour-cd4d956d9-9jzhb        1/1     Running   0          7m9s
flyte-deps-kubernetes-dashboard-8b7d858b7-lxs86   1/1     Running   0          7m9s
minio-67cb9c6dd7-p9l5k                            1/1     Running   0          7m9s
postgres-df78c4cfc-6k2lc                          1/1     Running   0          7m9s
Am I missing any step this time? Really appreciate it! 🙏
k

Kevin Su

07/04/2022, 8:43 AM
could you show the log message of 
flyte-deps-contour-envoy-qwp6d
by using (kubectl describe pods flyte-deps-contour-envoy-qwp6d)
sorry, I’m not familiar with k3s. I always use minikube or kind cluster.
t

Tan Li

07/04/2022, 9:17 AM
@Kevin Su Let me spin up a minikube cluster and try again, maybe’s it’s much easier to debug from there if I can have a working instance and then do the apple to apple config migration. Will come back to you on this, thanks!
Or just in case, is there any kind version deployment tutorial? I only saw a docker version but with K3s running within the docker on the project website
k

Kevin Su

07/04/2022, 10:37 AM
For kind deployment, below are the commands I ran
Copy code
kind create cluster --name flyte --config ~/kind-config --image kindest/node:v1.21.1
helm repo add flyteorg <https://helm.flyte.org>
helm install -n flyte flyte-deps flyteorg/flyte-deps --create-namespace -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-deps/values-sandbox.yaml>
helm install flyte flyteorg/flyte-core -n flyte -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-sandbox.yaml> --wait
My 
knd-config
file
Copy code
kind: Cluster
apiVersion: <http://kind.x-k8s.io/v1alpha4|kind.x-k8s.io/v1alpha4>
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30081
    hostPort: 30081
    protocol: TCP
  - containerPort: 30080
    hostPort: 30080
    protocol: TCP
  - containerPort: 30089
    hostPort: 30089
    protocol: TCP
  - containerPort: 32080
    hostPort: 32080
    protocol: TCP
  - containerPort: 30086
    hostPort: 30086
    protocol: TCP
  - containerPort: 30084
    hostPort: 30084
    protocol: TCP
  - containerPort: 30082
    hostPort: 30082
    protocol: TCP
  - containerPort: 30088
    hostPort: 30088
    protocol: TCP
👍 1
🙏 1
t

Tan Li

07/04/2022, 10:45 AM
Thanks @Kevin Su! This is very helpful!
19 Views
Powered by