My task that requires secrets management is curren...
# ask-the-community
k
My task that requires secrets management is currently stuck on queue state, likely due to certificate issues on webhook. Logs on Flyte propeller:
Copy code
plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": x509: certificate has expired or is not yet valid: current time 2023-02-01T05:22:42Z is after 2022-11-24T05:12:16Z
k
can you re-create the certificate
i think the default cert expires in 1 year
k
Yes, i would like to recreate the certificate, but i am not sure wha'ts the proper way to do it
k
restart the webhook pod?
cc @Eduardo Apolinario (eapolinario) / @jeev do you know?
k
I have tried restarting both the propeller and the webhook, but neither appears to help
In the secrets definition, i saw this:
Copy code
annotations:
    flyteLastUpdate: system-updated
    flyteUpdatedAt: 2021-11-24 05:12:18.699203733 +0000 UTC m=+4.56078966
So my assumption is, if i have done it correctly, the annotations on the secrets should be updated
Also, when i restarted the webhook, i saw this error message:
Copy code
2023/02/01 05:57:23 http: TLS handshake error from <ip address>:59692: remote error: tls: bad certificate
j
delete the secret and restart webhook should do it believe
it will create a new secret on restart, only if one doesnt already exist
k
Do i need to delete the mutating webhook configuration as well?
When i delete the secrets and restart the webhook, i still see TLS errors on the webhook pod
Despite the fact that new secrets are created
j
i think you're right
id imagine that it would be eventually consistent though
p
What’s strange is that the
ca.crt
in the
webhook
file system (
/etc/webhook/certs/ca.crt
) is different from the
ca.crt
in the
flyte-pod-webhook
secret which is mounted to the
webhook
. I tried to mount the same secret in other pod, but that issue is not happening. Deleting all (webhook deployment, secret, and mutatingwebhook config) and recreating it lead to same issue.
That is weird. So if you run init certs again it will not overwrite
p
Yeah, the k8s secret is new, but after mounted in the webhook the ca.crt is still old value. I am not sure where this ca.crt is coming from, it’s always having this enddate.
Copy code
▶ openssl x509 -enddate -noout -in cert_fs
notAfter=Nov 24 05:12:14 2022 GMT
Somehow it works after • delete flyte-pod-webhook secret • delete flyte-pod-webhook deployment • create empty flyte-pod-webhook secret ->
kubectl create secret generic flyte-pod-webhook
• recreate flyte-pod-webhook deployment Without the empty secret the deployment will fail with
Copy code
Warning  FailedMount  1s (x6 over 16s)  kubelet                                MountVolume.SetUp failed for volume "webhook-certs" : secret "flyte-pod-webhook" not found
s
Hey, we also faced the same problem and we fixed it by following what Aria suggested, is there any way we can fix it but without doing those manual steps?
k
this is odd and very important we fix
cc @Eduardo Apolinario (eapolinario) / @Yee lets capture this? @Stephen open to filing an issue?
cc @katrina note
s
Sure thing, we fixed it now but I can open an issue.
k
perfect, we will track it
s
Let me know if you want me to change things in the issue etc 🙂
nginx is using this https://github.com/kubernetes/ingress-nginx/tree/main/images/kube-webhook-certgen, don’t know if you find it interesting but thought I’d share anyway
120 Views