Has anyone else had issues with errors about pods being deleted externally?

[3/3] currentAttempt done. Last Error: SYSTEM::object [nerf-test-production/f3hmfc16u3esce-n0-0-n0-n0-0-n1-3] terminated in the background, manually

Its a bit difficult to know exactly what is going on but I don't think these pods are being deleted. Personally I think they are never actually getting created. This problem seems to happen a lot more when our k8s cluster is under high load. Tighter rate limiting in

kube-client-config

seems to help but I'm reluctant to reduce it further. Does anyone have any suggestions?

@Thomas Newton are you using

inject-finalizer

?

I don't see it in our config. Is it enabled by default?

Is it enabled by default?

I don't think so. You can set it under

configmap.k8s.plugins.k8s

inject-finalizer: true

Thanks. Would you mind explaining what it actually does?

It should inject a K8s finalizer at startTask. The finalizer will wait until the Task is complete to delete the Pod. It seems this is only a need when the K8s API server is under high load

Thanks I will give it a try.

sure, it's not a silver bullet but let's try 🙂

I think it solved the problem. Thanks for the help 🙂