https://flyte.org
Join Slack
My task that requires secrets management is currently stuck on queue state, likely due to certificat...
# flyte-support
j
My task that requires secrets management is currently stuck on queue state, likely due to certificate issues on webhook. Logs on Flyte propeller:
Copy code
plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": x509: certificate has expired or is not yet valid: current time 2023-02-01T05:22:42Z is after 2022-11-24T05:12:16Z
f
can you re-create the certificate
i think the default cert expires in 1 year
j
Yes, i would like to recreate the certificate, but i am not sure wha'ts the proper way to do it
f
restart the webhook pod?
cc @high-accountant-32689 / @freezing-boots-56761 do you know?
j
I have tried restarting both the propeller and the webhook, but neither appears to help
In the secrets definition, i saw this:
Copy code
annotations:
    flyteLastUpdate: system-updated
    flyteUpdatedAt: 2021-11-24 05:12:18.699203733 +0000 UTC m=+4.56078966
So my assumption is, if i have done it correctly, the annotations on the secrets should be updated
Also, when i restarted the webhook, i saw this error message:
Copy code
2023/02/01 05:57:23 http: TLS handshake error from <ip address>:59692: remote error: tls: bad certificate
f
delete the secret and restart webhook should do it believe
it will create a new secret on restart, only if one doesnt already exist
j
Do i need to delete the mutating webhook configuration as well?
When i delete the secrets and restart the webhook, i still see TLS errors on the webhook pod
Despite the fact that new secrets are created
f
i think you're right
id imagine that it would be eventually consistent though
m
What’s strange is that the 
ca.crt
in the 
webhook
file system (
/etc/webhook/certs/ca.crt
) is different from the 
ca.crt
in the 
flyte-pod-webhook
secret which is mounted to the 
webhook
. I tried to mount the same secret in other pod, but that issue is not happening. Deleting all (webhook deployment, secret, and mutatingwebhook config) and recreating it lead to same issue.
f
That is weird. So if you run init certs again it will not overwrite
m
Yeah, the k8s secret is new, but after mounted in the webhook the ca.crt is still old value. I am not sure where this ca.crt is coming from, it’s always having this enddate.
Copy code
▶ openssl x509 -enddate -noout -in cert_fs
notAfter=Nov 24 05:12:14 2022 GMT
Somehow it works after • delete flyte-pod-webhook secret • delete flyte-pod-webhook deployment • create empty flyte-pod-webhook secret -> 
kubectl create secret generic flyte-pod-webhook
• recreate flyte-pod-webhook deployment Without the empty secret the deployment will fail with
Copy code
Warning  FailedMount  1s (x6 over 16s)  kubelet                                MountVolume.SetUp failed for volume "webhook-certs" : secret "flyte-pod-webhook" not found
j
Hey, we also faced the same problem and we fixed it by following what Aria suggested, is there any way we can fix it but without doing those manual steps?
f
this is odd and very important we fix
cc @high-accountant-32689 / @thankful-minister-83577 lets capture this? @jolly-whale-9142 open to filing an issue?
cc @acceptable-policeman-57188 note
👀 1
j
Sure thing, we fixed it now but I can open an issue.
f
perfect, we will track it
j
Let me know if you want me to change things in the issue etc 🙂
nginx is using this https://github.com/kubernetes/ingress-nginx/tree/main/images/kube-webhook-certgen, don’t know if you find it interesting but thought I’d share anyway
162 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by