Hi team.. I’m suddenly running into a weird error ...
# flyte-supportl
little-cricket-84530
08/29/2023, 3:22 AMHi team.. I’m suddenly running into a weird error with a map task (3rd time now). Some of the map tasks fail with this error and eventually the whole workflow fails. Nothing has changed as far as our config or workflow code is concerned. The task has
interruptible=False Copy code
code:"UnexpectedObjectDeletion" message:"object [<k8s-namespace-name>/a5vmlnn7xxvpbjv5qztd-n6-0-33] terminated in the background, manuallylittle-cricket-84530
08/29/2023, 3:26 AMAlso are successful map tasks not cached? There were 20 that succeeded out of 50 but looks like on a recover/relaunch they are running all over again
f
freezing-airport-6809
08/29/2023, 3:56 AM
Turn on inject-finalizer - you are running a lot of pods and they are being deleted by k8s
l
little-cricket-84530
08/29/2023, 4:14 AMThanks.. let me try this. It’s curious since this workflow has been running for several months now\
little-cricket-84530
08/29/2023, 4:15 AMLooks like we already have it set to true for copilot
little-cricket-84530
08/29/2023, 4:16 AMDo the propeller and scheduler also need it?
f
freezing-airport-6809
08/29/2023, 4:54 AM
ohh you need it for propeller
freezing-airport-6809
08/29/2023, 4:54 AM
what is for co-pilot
freezing-airport-6809
08/29/2023, 4:54 AM
that is not needed
l
little-cricket-84530
08/29/2023, 11:18 PMThis is a screenshot of the yaml from flyte example setup…
So should the k8s plugin config with inject finalizer be under core -> propeller -> plugins -> k8s
100 Views