victorious-branch-8796
06/30/2022, 7:17 AMsvclb-flyte-deps-contour-envoy-c519e84f-lnpvm
keeps pending in the namespace kube-system. I worked around this by upgradeing the flyte-deps config.
However, I am still confronting the envoy pod pending issue. Terminal output are in the reply thread. Anyone know how should I fix this? In general, it seems to be some port allocation issue, but I really don’t have any idea how to debug this futher, lol. Really appreciate it! 🙏victorious-branch-8796
06/30/2022, 7:18 AM$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-6c79684f77-2t6g4 1/1 Running 0 77m
coredns-d76bd69b-cnh8f 1/1 Running 0 77m
svclb-traefik-d46b5b65-pck27 2/2 Running 0 77m
traefik-df4ff85d6-pn7xs 1/1 Running 0 77m
metrics-server-7cd5fcb6b7-f4c8q 1/1 Running 0 77m
svclb-minio-3bbacdee-whhs6 2/2 Running 0 16m
svclb-flyte-deps-kubernetes-dashboard-e2d186b7-xxq8x 1/1 Running 0 16m
svclb-flyte-deps-contour-envoy-c519e84f-lnpvm 2/2 Running 0 12m
$ kubectl get pods -n flyte
NAME READY STATUS RESTARTS AGE
flyte-deps-contour-envoy-k8rmw 0/2 Pending 0 18m
postgres-df78c4cfc-8q6sz 1/1 Running 0 18m
flyte-deps-kubernetes-dashboard-8b7d858b7-78tgn 1/1 Running 0 18m
minio-67cb9c6dd7-rhppm 1/1 Running 0 18m
flyte-deps-contour-contour-cd4d956d9-kkl2w 1/1 Running 0 18m
flyteconsole-845f5c44b-tmpsr 1/1 Running 0 10m
syncresources-766786b498-rv5jx 1/1 Running 0 10m
flytepropeller-8645d94f64-b9pzd 1/1 Running 0 10m
datacatalog-9c879977b-chrmw 1/1 Running 0 10m
flyte-pod-webhook-5bd8f7fcb8-f9tgm 1/1 Running 0 10m
flyteadmin-55d8886bd6-cvmf4 1/1 Running 0 10m
I then inspect the envoy pod in namespace flyte by doing kubectl describe, the here is the output
Name: flyte-deps-contour-envoy-k8rmw
Namespace: flyte
Priority: 0
Node: <none>
Labels: <http://app.kubernetes.io/component=envoy|app.kubernetes.io/component=envoy>
<http://app.kubernetes.io/instance=flyte-deps|app.kubernetes.io/instance=flyte-deps>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=contour|app.kubernetes.io/name=contour>
controller-revision-hash=67bdb7bd55
<http://helm.sh/chart=contour-7.10.1|helm.sh/chart=contour-7.10.1>
pod-template-generation=1
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: DaemonSet/flyte-deps-contour-envoy
Init Containers:
envoy-initconfig:
Image: <http://docker.io/bitnami/contour:1.20.1-debian-10-r53|docker.io/bitnami/contour:1.20.1-debian-10-r53>
Port: <none>
Host Port: <none>
Command:
contour
Args:
bootstrap
/config/envoy.json
--xds-address=flyte-deps-contour
--xds-port=8001
--resources-dir=/config/resources
--envoy-cafile=/certs/ca.crt
--envoy-cert-file=/certs/tls.crt
--envoy-key-file=/certs/tls.key
Limits:
cpu: 100m
memory: 100Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
CONTOUR_NAMESPACE: flyte (v1:metadata.namespace)
Mounts:
/admin from envoy-admin (rw)
/certs from envoycert (ro)
/config from envoy-config (rw)
Containers:
shutdown-manager:
Image: <http://docker.io/bitnami/contour:1.20.1-debian-10-r53|docker.io/bitnami/contour:1.20.1-debian-10-r53>
Port: <none>
Host Port: <none>
Command:
contour
Args:
envoy
shutdown-manager
Liveness: http-get http://:8090/healthz delay=120s timeout=5s period=20s #success=1 #failure=6
Environment: <none>
Mounts:
/admin from envoy-admin (rw)
envoy:
Image: <http://docker.io/bitnami/envoy:1.21.1-debian-10-r55|docker.io/bitnami/envoy:1.21.1-debian-10-r55>
Ports: 8080/TCP, 8443/TCP, 8002/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Command:
envoy
Args:
-c
/config/envoy.json
--service-cluster $(CONTOUR_NAMESPACE)
--service-node $(ENVOY_POD_NAME)
--log-level info
Limits:
cpu: 100m
memory: 100Mi
Requests:
cpu: 10m
memory: 50Mi
Liveness: http-get http://:8002/ready delay=120s timeout=5s period=20s #success=1 #failure=6
Readiness: http-get http://:8002/ready delay=10s timeout=1s period=3s #success=1 #failure=3
Environment:
CONTOUR_NAMESPACE: flyte (v1:metadata.namespace)
ENVOY_POD_NAME: flyte-deps-contour-envoy-k8rmw (v1:metadata.name)
Mounts:
/admin from envoy-admin (rw)
/certs from envoycert (rw)
/config from envoy-config (rw)
Conditions:
Type Status
PodScheduled False
Volumes:
envoy-admin:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
envoy-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
envoycert:
Type: Secret (a volume populated by a Secret)
SecretName: envoycert
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists
<http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists
<http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule> op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m41s default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
Warning FailedScheduling 5m36s (x1 over 5m39s) default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
victorious-branch-8796
06/30/2022, 7:47 AMenvoy:
Image: <http://docker.io/bitnami/envoy:1.21.1-debian-10-r55|docker.io/bitnami/envoy:1.21.1-debian-10-r55>
Ports: 8080/TCP, 8443/TCP, 8002/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Command:
envoy
glamorous-carpet-83516
06/30/2022, 8:26 AMvictorious-branch-8796
06/30/2022, 8:37 AMglamorous-carpet-83516
06/30/2022, 9:28 AM--no-deploy=traefik
. https://github.com/flyteorg/flyte/blob/b16ffd76934d690068db1265ac9907a278fba2ee/docker/sandbox/flyte-entrypoint-default.sh#L34glamorous-carpet-83516
06/30/2022, 9:32 AMglamorous-carpet-83516
06/30/2022, 9:32 AMfreezing-airport-6809
victorious-branch-8796
07/04/2022, 2:13 AMhelm repo add flyteorg <https://helm.flyte.org>
helm install -n flyte flyte-deps flyteorg/flyte-deps --create-namespace -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-deps/values-sandbox.yaml>
helm install flyte flyteorg/flyte-core -n flyte -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-sandbox.yaml> --wait
When I call kubectl get pods -n flyte
, the envoy proxy is still pending.
NAME READY STATUS RESTARTS AGE
flyte-deps-contour-envoy-qwp6d 0/2 Pending 0 7m9s
flyte-deps-contour-contour-cd4d956d9-9jzhb 1/1 Running 0 7m9s
flyte-deps-kubernetes-dashboard-8b7d858b7-lxs86 1/1 Running 0 7m9s
minio-67cb9c6dd7-p9l5k 1/1 Running 0 7m9s
postgres-df78c4cfc-6k2lc 1/1 Running 0 7m9s
Am I missing any step this time? Really appreciate it! 🙏glamorous-carpet-83516
07/04/2022, 8:43 AMflyte-deps-contour-envoy-qwp6d
by using (kubectl describe pods flyte-deps-contour-envoy-qwp6d)glamorous-carpet-83516
07/04/2022, 8:46 AMvictorious-branch-8796
07/04/2022, 9:17 AMvictorious-branch-8796
07/04/2022, 9:19 AMglamorous-carpet-83516
07/04/2022, 10:37 AMkind create cluster --name flyte --config ~/kind-config --image kindest/node:v1.21.1
helm repo add flyteorg <https://helm.flyte.org>
helm install -n flyte flyte-deps flyteorg/flyte-deps --create-namespace -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-deps/values-sandbox.yaml>
helm install flyte flyteorg/flyte-core -n flyte -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-sandbox.yaml> --wait
My knd-config
file
kind: Cluster
apiVersion: <http://kind.x-k8s.io/v1alpha4|kind.x-k8s.io/v1alpha4>
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30081
hostPort: 30081
protocol: TCP
- containerPort: 30080
hostPort: 30080
protocol: TCP
- containerPort: 30089
hostPort: 30089
protocol: TCP
- containerPort: 32080
hostPort: 32080
protocol: TCP
- containerPort: 30086
hostPort: 30086
protocol: TCP
- containerPort: 30084
hostPort: 30084
protocol: TCP
- containerPort: 30082
hostPort: 30082
protocol: TCP
- containerPort: 30088
hostPort: 30088
protocol: TCP
victorious-branch-8796
07/04/2022, 10:45 AM