Hi team.. I’m suddenly running into a weird error ...
# ask-the-communityr
Rupsha Chaudhuri
08/29/2023, 3:22 AMHi team.. I’m suddenly running into a weird error with a map task (3rd time now). Some of the map tasks fail with this error and eventually the whole workflow fails. Nothing has changed as far as our config or workflow code is concerned. The task has
interruptible=False Copy code
code:"UnexpectedObjectDeletion" message:"object [<k8s-namespace-name>/a5vmlnn7xxvpbjv5qztd-n6-0-33] terminated in the background, manuallyRupsha Chaudhuri
08/29/2023, 3:26 AMAlso are successful map tasks not cached? There were 20 that succeeded out of 50 but looks like on a recover/relaunch they are running all over again
k
Ketan (kumare3)
08/29/2023, 3:56 AM
Turn on inject-finalizer - you are running a lot of pods and they are being deleted by k8s
r
Rupsha Chaudhuri
08/29/2023, 4:14 AMThanks.. let me try this. It’s curious since this workflow has been running for several months now\
Rupsha Chaudhuri
08/29/2023, 4:15 AMLooks like we already have it set to true for copilot
Rupsha Chaudhuri
08/29/2023, 4:16 AMDo the propeller and scheduler also need it?
k
Ketan (kumare3)
08/29/2023, 4:54 AM
ohh you need it for propeller
Ketan (kumare3)
08/29/2023, 4:54 AM
what is for co-pilot
Ketan (kumare3)
08/29/2023, 4:54 AM
that is not needed
r
Rupsha Chaudhuri
08/29/2023, 11:18 PMThis is a screenshot of the yaml from flyte example setup…
So should the k8s plugin config with inject finalizer be under core -> propeller -> plugins -> k8s
52 Views