https://flyte.org
Join Slack
Hi team.. I’m suddenly running into a weird error with a map task (3rd time now). Some of the map ta...
# flyte-support
l
Hi team.. I’m suddenly running into a weird error with a map task (3rd time now). Some of the map tasks fail with this error and eventually the whole workflow fails. Nothing has changed as far as our config or workflow code is concerned. The task has 
interruptible=False
Copy code
code:"UnexpectedObjectDeletion" message:"object [<k8s-namespace-name>/a5vmlnn7xxvpbjv5qztd-n6-0-33] terminated in the background, manually
Also are successful map tasks not cached? There were 20 that succeeded out of 50 but looks like on a recover/relaunch they are running all over again
f
Turn on inject-finalizer - you are running a lot of pods and they are being deleted by k8s
l
Thanks.. let me try this. It’s curious since this workflow has been running for several months now\
Looks like we already have it set to true for copilot
Do the propeller and scheduler also need it?
f
ohh you need it for propeller
what is for co-pilot
that is not needed
l
This is a screenshot of the yaml from flyte example setup… So should the k8s plugin config with inject finalizer be under core -> propeller -> plugins -> k8s
121 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by