Hi Flyte team, We see this issue with our flyte-pod-webhook. The issues goes way if restart the deployment.

kubectl logs flyte-pod-webhook-57cff5dd75-w596k
Defaulted container "webhook" out of: webhook, generate-secrets (init)
time="2022-09-14T22:11:09Z" level=info msg=------------------------------------------------------------------------
time="2022-09-14T22:11:09Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-09-14 22:11:09.195785843 +0000 UTC m=+0.480189926]"
time="2022-09-14T22:11:09Z" level=info msg=------------------------------------------------------------------------
time="2022-09-14T22:11:09Z" level=info msg="Detected: 64 CPU's\n"
{"json":{},"level":"fatal","msg":"Failed to create controller manager. Error: failed to initialize controller-runtime manager: error listening on :8080: listen tcp :8080: bind: address already in use","ts":"2022-09-14T22:11:13Z"}

the certificate expired?

Dan filed an issue already: https://github.com/flyteorg/flyte/issues/2871

@Ketan (kumare3) where are we seeing an expired cert? @Fredrick can you provide a little more context? Is this a deployment using the helm chart? Did this occur on the initial deployment or sometime after?

We definitely have not had this running for a year, so I don’t think this is a case of the certificate expiring

ohh sorry i assumed this was the “Frederick” from some other company. I see it now, bind address already in use

Hello… any updates? no pressure 🙂

Cc @Dan Rammer (hamersaw) can you look into this

@Rupsha Chaudhuri I was unable to repro this locally. Has this occurred more than once? Can I get a little more context? maybe dump the pod yaml?

@Fredrick ^

@Fredrick Has this occurred more than once? just on the initial startup or after awhile?

@Dan Rammer (hamersaw) thank you for looking- @Rupsha Chaudhuri we willl take a look, is this blocking you?

No its not a blocker. It does not happen very often.

This is now giving me a lot of grief.. I couldn’t get any workflows to succeed in the last couple of days even after restarting the pod multiple times 😞

@Rupsha Chaudhuri, so sorry about this. I will take a deep dive here immediately. Will update accordingly.

In case it’s relevant, @Fredrick mentioned that in our deployment we have

hostNetwork: true

without which web-hooks don’t work on EKS

So if the

hostNetwork: true

this means the Pod will have access to the host network. In this case it means that what is probably happening is something else is bound to :8080 on that host. This also explains why restarting the Pod sometimes helps, k8s schedules it to another host which has 8080 open. I'm wondering if there is another solution to enable networking here that will mitigate these issues - pinging internally.

yes we are using EKS with calico CNI. I understand with hostNetwork there can be conflict the ports. But why is flyte-pod-webhook listening on 8080 ? i thought it used 9443

OK, so was able to dive into this and figure it out a bit. So the 9443 is the pod webhook listener, you are correct - where k8s calls to update the Pod definition during secrets. The 8080 port is what the k8s controller-runtime manager metrics server binds to by default. The controller-runtime manager is started in the pod-webhook because it needs to have an informer, cache, etc with the k8s api server to work correctly. For an immediate fix, we can set the metrics server port to 0 (which disables it). However, this will require a change to the FlytePropeller codebase. We're hoping to have this out out quickly (by tomorrow) and it should alleviate this issue you're having. As an end goal, by disabling the metrics server we loose access to some k8s prometheus metrics, we would like to add these metrics to our prometheus metrics server. This will require a little more work, but is certainly possible.

Yes we can disable controller-runtime manager's metrics server. Let us know when we have this ability to do that.

@Fredrick I just opened a PR to disable this, once it's merged (will push through quickly) we will automatically build a new flytepropeller image that can be used. I will update once it's ready.

@Fredrick and @Rupsha Chaudhuri thank you for the patience and helping us find a potential bug

@Fredrick @Rupsha Chaudhuri OK we merged the PR to disable the k8s controller-runtime metrics server, the newer FlytePropeller image is "cr.flyte.org/flyteorg/flytepropeller:v1.1.42" as denoted by the release. You can update the flyte-pod-webhook deployment with this new image and should be able to verify that the pod does not open port 8080. Please let us know if you run into any problems!

Thank you so much.. We’ll keep you posted

@Dan Rammer (hamersaw) after upgrading to v1.1.42, the flyte-pod-webhook is not coming up

kubectl logs flyte-pod-webhook-587d7d5977-lxwwb
Defaulted container "webhook" out of: webhook, generate-secrets (init)
time="2022-10-14T17:29:32Z" level=info msg=------------------------------------------------------------------------
time="2022-10-14T17:29:32Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-10-14 17:29:32.394880612 +0000 UTC m=+0.247511472]"
time="2022-10-14T17:29:32Z" level=info msg=------------------------------------------------------------------------
time="2022-10-14T17:29:32Z" level=info msg="Detected: 64 CPU's\n"
{"metrics-prefix":"flyte:","certDir":"/etc/webhook/certs","localCert":false,"listenPort":9443,"serviceName":"flyte-pod-webhook","servicePort":443,"secretName":"flyte-pod-webhook","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
{"json":{},"level":"error","msg":"Failed to start profiling server. Error: listen tcp :10254: bind: address already in use","ts":"2022-10-14T17:29:34Z"}
{"json":{},"level":"fatal","msg":"Failed to Start profiling and metrics server. Error: failed to start profiling server, listen tcp :10254: bind: address already in use","ts":"2022-10-14T17:29:34Z"}

so this should be a little easier to reconcile. 10254 is the default port that FlytePropeller uses to serve metrics. so this is opened in both FlytePropeller pod and the flyte-pod-webhook pod. You can update this in the configuration in the propeller configmap with this value. however, i do believe that the flytepropeller pod and the flyte-pod-webhook pod use the same configmap by default, so to enforce two separate values you probably need to make a separate configmap for the flyte-pod-webhook.

is it possible to disable profiler ?

So currently that is not implemented, but I would support it as an option. If you filed an issue it would be a pretty quick change. This would be a great community contribution.

created a ticket https://github.com/flyteorg/flyte/issues/2985