https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
k

Khor Shu Heng

02/01/2023, 5:27 AM
My task that requires secrets management is currently stuck on queue state, likely due to certificate issues on webhook. Logs on Flyte propeller:
Copy code
plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": x509: certificate has expired or is not yet valid: current time 2023-02-01T05:22:42Z is after 2022-11-24T05:12:16Z
k

Ketan (kumare3)

02/01/2023, 5:48 AM
can you re-create the certificate
i think the default cert expires in 1 year
k

Khor Shu Heng

02/01/2023, 5:48 AM
Yes, i would like to recreate the certificate, but i am not sure wha'ts the proper way to do it
k

Ketan (kumare3)

02/01/2023, 5:48 AM
restart the webhook pod?
cc @Eduardo Apolinario (eapolinario) / @jeev do you know?
k

Khor Shu Heng

02/01/2023, 5:58 AM
I have tried restarting both the propeller and the webhook, but neither appears to help
In the secrets definition, i saw this:
Copy code
annotations:
    flyteLastUpdate: system-updated
    flyteUpdatedAt: 2021-11-24 05:12:18.699203733 +0000 UTC m=+4.56078966
So my assumption is, if i have done it correctly, the annotations on the secrets should be updated
Also, when i restarted the webhook, i saw this error message:
Copy code
2023/02/01 05:57:23 http: TLS handshake error from <ip address>:59692: remote error: tls: bad certificate
j

jeev

02/01/2023, 6:08 AM
delete the secret and restart webhook should do it believe
it will create a new secret on restart, only if one doesnt already exist
k

Khor Shu Heng

02/01/2023, 6:11 AM
Do i need to delete the mutating webhook configuration as well?
When i delete the secrets and restart the webhook, i still see TLS errors on the webhook pod
Despite the fact that new secrets are created
j

jeev

02/01/2023, 6:12 AM
i think you're right
id imagine that it would be eventually consistent though
p

Pradithya Aria Pura

02/02/2023, 6:30 AM
What’s strange is that the 
ca.crt
in the 
webhook
file system (
/etc/webhook/certs/ca.crt
) is different from the 
ca.crt
in the 
flyte-pod-webhook
secret which is mounted to the 
webhook
. I tried to mount the same secret in other pod, but that issue is not happening. Deleting all (webhook deployment, secret, and mutatingwebhook config) and recreating it lead to same issue.
k

Ketan (kumare3)

02/02/2023, 6:41 AM
That is weird. So if you run init certs again it will not overwrite
p

Pradithya Aria Pura

02/02/2023, 6:44 AM
Yeah, the k8s secret is new, but after mounted in the webhook the ca.crt is still old value. I am not sure where this ca.crt is coming from, it’s always having this enddate.
Copy code
▶ openssl x509 -enddate -noout -in cert_fs
notAfter=Nov 24 05:12:14 2022 GMT
Somehow it works after • delete flyte-pod-webhook secret • delete flyte-pod-webhook deployment • create empty flyte-pod-webhook secret -> 
kubectl create secret generic flyte-pod-webhook
• recreate flyte-pod-webhook deployment Without the empty secret the deployment will fail with
Copy code
Warning  FailedMount  1s (x6 over 16s)  kubelet                                MountVolume.SetUp failed for volume "webhook-certs" : secret "flyte-pod-webhook" not found
s

Stephen

02/28/2023, 1:19 PM
Hey, we also faced the same problem and we fixed it by following what Aria suggested, is there any way we can fix it but without doing those manual steps?
k

Ketan (kumare3)

02/28/2023, 6:08 PM
this is odd and very important we fix
cc @Eduardo Apolinario (eapolinario) / @Yee lets capture this? @Stephen open to filing an issue?
cc @katrina note
s

Stephen

02/28/2023, 6:09 PM
Sure thing, we fixed it now but I can open an issue.
k

Ketan (kumare3)

02/28/2023, 6:09 PM
perfect, we will track it
s

Stephen

02/28/2023, 6:23 PM
Let me know if you want me to change things in the issue etc 🙂
nginx is using this https://github.com/kubernetes/ingress-nginx/tree/main/images/kube-webhook-certgen, don’t know if you find it interesting but thought I’d share anyway
Powered by