https://flyte.org logo
#ask-the-community
Title
# ask-the-community
t

Thomas Newton

11/21/2023, 2:26 PM
Has anyone else had issues with errors about pods being deleted externally?
Copy code
[3/3] currentAttempt done. Last Error: SYSTEM::object [nerf-test-production/f3hmfc16u3esce-n0-0-n0-n0-0-n1-3] terminated in the background, manually
Its a bit difficult to know exactly what is going on but I don't think these pods are being deleted. Personally I think they are never actually getting created. This problem seems to happen a lot more when our k8s cluster is under high load. Tighter rate limiting in
kube-client-config
seems to help but I'm reluctant to reduce it further. Does anyone have any suggestions?
d

David Espejo (he/him)

11/21/2023, 4:27 PM
@Thomas Newton are you using
inject-finalizer
?
t

Thomas Newton

11/21/2023, 4:56 PM
I don't see it in our config. Is it enabled by default?
d

David Espejo (he/him)

11/21/2023, 5:05 PM
Is it enabled by default?
I don't think so. You can set it under
configmap.k8s.plugins.k8s
inject-finalizer: true
t

Thomas Newton

11/21/2023, 5:05 PM
Thanks. Would you mind explaining what it actually does?
d

David Espejo (he/him)

11/21/2023, 5:08 PM
It should inject a K8s finalizer at startTask. The finalizer will wait until the Task is complete to delete the Pod. It seems this is only a need when the K8s API server is under high load
t

Thomas Newton

11/21/2023, 5:08 PM
Thanks I will give it a try.
d

David Espejo (he/him)

11/21/2023, 5:09 PM
sure, it's not a silver bullet but let's try 🙂
t

Thomas Newton

11/29/2023, 5:57 PM
I think it solved the problem. Thanks for the help 🙂