Fredrick
09/14/2022, 10:17 PMkubectl logs flyte-pod-webhook-57cff5dd75-w596k
Defaulted container "webhook" out of: webhook, generate-secrets (init)
time="2022-09-14T22:11:09Z" level=info msg=------------------------------------------------------------------------
time="2022-09-14T22:11:09Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-09-14 22:11:09.195785843 +0000 UTC m=+0.480189926]"
time="2022-09-14T22:11:09Z" level=info msg=------------------------------------------------------------------------
time="2022-09-14T22:11:09Z" level=info msg="Detected: 64 CPU's\n"
{"json":{},"level":"fatal","msg":"Failed to create controller manager. Error: failed to initialize controller-runtime manager: error listening on :8080: listen tcp :8080: bind: address already in use","ts":"2022-09-14T22:11:13Z"}
Ketan (kumare3)
Samhita Alla
Dan Rammer (hamersaw)
09/15/2022, 3:06 PMRupsha Chaudhuri
09/15/2022, 8:11 PMKetan (kumare3)
Rupsha Chaudhuri
09/21/2022, 5:27 PMKetan (kumare3)
Dan Rammer (hamersaw)
09/21/2022, 6:01 PMRupsha Chaudhuri
09/21/2022, 6:01 PMDan Rammer (hamersaw)
09/21/2022, 7:00 PMKetan (kumare3)
Fredrick
09/23/2022, 5:50 PMRupsha Chaudhuri
10/10/2022, 4:09 PMDan Rammer (hamersaw)
10/10/2022, 4:25 PM`kubectl -n flyte exec flyte-pod-webhook-788c4c876c-6f58f -- netstat -tulpn
On my deployment locally - it shows propeller (ie. pod-webhook) is correctly bound to 8080. It may be useful to see this output both on the defauted webhook
container and on the generate-secrets
init container. So something like:
hamersaw@ragnarok:~$ kubectl -n flyte exec flyte-pod-webhook-788c4c876c-6f58f -- netstat -tulpn
Defaulted container "webhook" out of: webhook, generate-secrets (init)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::9443 :::* LISTEN 1/flytepropeller
tcp 0 0 :::10254 :::* LISTEN 1/flytepropeller
tcp 0 0 :::8080 :::* LISTEN 1/flytepropeller
Rupsha Chaudhuri
10/10/2022, 5:01 PMhostNetwork: true
without which web-hooks don’t work on EKSDan Rammer (hamersaw)
10/10/2022, 5:13 PMhostNetwork: true
this means the Pod will have access to the host network. In this case it means that what is probably happening is something else is bound to :8080 on that host. This also explains why restarting the Pod sometimes helps, k8s schedules it to another host which has 8080 open. I'm wondering if there is another solution to enable networking here that will mitigate these issues - pinging internally.Fredrick
10/10/2022, 6:13 PMDan Rammer (hamersaw)
10/10/2022, 7:08 PMFredrick
10/10/2022, 7:24 PMDan Rammer (hamersaw)
10/10/2022, 11:17 PMKetan (kumare3)
Dan Rammer (hamersaw)
10/12/2022, 4:59 PMRupsha Chaudhuri
10/12/2022, 5:00 PMFredrick
10/14/2022, 5:31 PMkubectl logs flyte-pod-webhook-587d7d5977-lxwwb
Defaulted container "webhook" out of: webhook, generate-secrets (init)
time="2022-10-14T17:29:32Z" level=info msg=------------------------------------------------------------------------
time="2022-10-14T17:29:32Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-10-14 17:29:32.394880612 +0000 UTC m=+0.247511472]"
time="2022-10-14T17:29:32Z" level=info msg=------------------------------------------------------------------------
time="2022-10-14T17:29:32Z" level=info msg="Detected: 64 CPU's\n"
{"metrics-prefix":"flyte:","certDir":"/etc/webhook/certs","localCert":false,"listenPort":9443,"serviceName":"flyte-pod-webhook","servicePort":443,"secretName":"flyte-pod-webhook","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
{"json":{},"level":"error","msg":"Failed to start profiling server. Error: listen tcp :10254: bind: address already in use","ts":"2022-10-14T17:29:34Z"}
{"json":{},"level":"fatal","msg":"Failed to Start profiling and metrics server. Error: failed to start profiling server, listen tcp :10254: bind: address already in use","ts":"2022-10-14T17:29:34Z"}
Dan Rammer (hamersaw)
10/14/2022, 5:56 PMFredrick
10/14/2022, 6:57 PMDan Rammer (hamersaw)
10/14/2022, 7:02 PMFredrick
10/14/2022, 8:59 PM