adamant-sandwich-39749
09/14/2022, 10:17 PMkubectl logs flyte-pod-webhook-57cff5dd75-w596k
Defaulted container "webhook" out of: webhook, generate-secrets (init)
time="2022-09-14T22:11:09Z" level=info msg=------------------------------------------------------------------------
time="2022-09-14T22:11:09Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-09-14 22:11:09.195785843 +0000 UTC m=+0.480189926]"
time="2022-09-14T22:11:09Z" level=info msg=------------------------------------------------------------------------
time="2022-09-14T22:11:09Z" level=info msg="Detected: 64 CPU's\n"
{"json":{},"level":"fatal","msg":"Failed to create controller manager. Error: failed to initialize controller-runtime manager: error listening on :8080: listen tcp :8080: bind: address already in use","ts":"2022-09-14T22:11:13Z"}
freezing-airport-6809
freezing-airport-6809
tall-lock-23197
hallowed-mouse-14616
09/15/2022, 3:06 PMlittle-cricket-84530
09/15/2022, 8:11 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
little-cricket-84530
09/21/2022, 5:27 PMfreezing-airport-6809
hallowed-mouse-14616
09/21/2022, 6:01 PMlittle-cricket-84530
09/21/2022, 6:01 PMadamant-sandwich-39749
09/21/2022, 6:39 PMhallowed-mouse-14616
09/21/2022, 7:00 PMhallowed-mouse-14616
09/23/2022, 1:24 PMfreezing-airport-6809
adamant-sandwich-39749
09/23/2022, 5:50 PMlittle-cricket-84530
10/10/2022, 4:09 PMlittle-cricket-84530
10/10/2022, 4:19 PMlittle-cricket-84530
10/10/2022, 4:19 PMhallowed-mouse-14616
10/10/2022, 4:25 PMhallowed-mouse-14616
10/10/2022, 4:46 PMhallowed-mouse-14616
10/10/2022, 4:56 PMhallowed-mouse-14616
10/10/2022, 5:01 PM`kubectl -n flyte exec flyte-pod-webhook-788c4c876c-6f58f -- netstat -tulpn
On my deployment locally - it shows propeller (ie. pod-webhook) is correctly bound to 8080. It may be useful to see this output both on the defauted webhook
container and on the generate-secrets
init container. So something like:
hamersaw@ragnarok:~$ kubectl -n flyte exec flyte-pod-webhook-788c4c876c-6f58f -- netstat -tulpn
Defaulted container "webhook" out of: webhook, generate-secrets (init)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::9443 :::* LISTEN 1/flytepropeller
tcp 0 0 :::10254 :::* LISTEN 1/flytepropeller
tcp 0 0 :::8080 :::* LISTEN 1/flytepropeller
little-cricket-84530
10/10/2022, 5:01 PMhostNetwork: true
without which web-hooks don’t work on EKSlittle-cricket-84530
10/10/2022, 5:04 PMhallowed-mouse-14616
10/10/2022, 5:13 PMhostNetwork: true
this means the Pod will have access to the host network. In this case it means that what is probably happening is something else is bound to :8080 on that host. This also explains why restarting the Pod sometimes helps, k8s schedules it to another host which has 8080 open. I'm wondering if there is another solution to enable networking here that will mitigate these issues - pinging internally.hallowed-mouse-14616
10/10/2022, 5:20 PMadamant-sandwich-39749
10/10/2022, 6:13 PMhallowed-mouse-14616
10/10/2022, 7:08 PMhallowed-mouse-14616
10/10/2022, 7:09 PMadamant-sandwich-39749
10/10/2022, 7:24 PMhallowed-mouse-14616
10/10/2022, 11:17 PMhallowed-mouse-14616
10/10/2022, 11:17 PMfreezing-airport-6809
hallowed-mouse-14616
10/12/2022, 4:59 PMlittle-cricket-84530
10/12/2022, 5:00 PMadamant-sandwich-39749
10/14/2022, 5:31 PMkubectl logs flyte-pod-webhook-587d7d5977-lxwwb
Defaulted container "webhook" out of: webhook, generate-secrets (init)
time="2022-10-14T17:29:32Z" level=info msg=------------------------------------------------------------------------
time="2022-10-14T17:29:32Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-10-14 17:29:32.394880612 +0000 UTC m=+0.247511472]"
time="2022-10-14T17:29:32Z" level=info msg=------------------------------------------------------------------------
time="2022-10-14T17:29:32Z" level=info msg="Detected: 64 CPU's\n"
{"metrics-prefix":"flyte:","certDir":"/etc/webhook/certs","localCert":false,"listenPort":9443,"serviceName":"flyte-pod-webhook","servicePort":443,"secretName":"flyte-pod-webhook","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
{"json":{},"level":"error","msg":"Failed to start profiling server. Error: listen tcp :10254: bind: address already in use","ts":"2022-10-14T17:29:34Z"}
{"json":{},"level":"fatal","msg":"Failed to Start profiling and metrics server. Error: failed to start profiling server, listen tcp :10254: bind: address already in use","ts":"2022-10-14T17:29:34Z"}
hallowed-mouse-14616
10/14/2022, 5:56 PMadamant-sandwich-39749
10/14/2022, 6:57 PMadamant-sandwich-39749
10/14/2022, 6:58 PMhallowed-mouse-14616
10/14/2022, 7:02 PMadamant-sandwich-39749
10/14/2022, 8:59 PM