Hi team.. we are occasionally seeing this error with a map task… rerun succeeds

RuntimeExecutionError: max number of system retry attempts [31/30] exhausted. Last known status message: failed at Node[dn0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [k8s-array]: 
Operation cannot be fulfilled on pods "f82e2be5d27cc00fc000-n7-0-dn0-0-2427": the object has been modified; please apply your changes to the latest version and try again

Flyte backend: 1.13.1

It seems finalizer is missing or was this a bug that was fixed I think in object deletion

umm.. so should we update to a later version?

We put in a ~fix for what seems to be this issue that was released as part of 1.13.1. Just wanted to confirm that your propeller version is >= 1.13.1. There are a few configs you can try adjusting to alleviate this issue but those adjustments hopefully shouldn’t be needed with that change.

Confirmed that propeller is on 1.13.1

oh wait, it looks like you're using legacy map tasks. That's why that change isn't resolving the issue as we didn't update the clearFinalizers method utilized by the legacy map task's plugin. What version of flytekit are you running? In v1.12.0 array node map tasks became the default for map tasks. There shouldn't be a need for any changes to workflow code if/when you update flytekit to utilize the new map task implementation.

Ah.. I’m on 1.11.. switch to 1.12? Any code changes required with that?

yup - shouldn't need any code changes for map tasks for versions after 1.12. I think the latest is v1.13.11

Update went smoothly.. AND the logs link for map tasks is finally working properly