Hi everyone! 👋 We’re seeing various tasks fail with

terminated in the background, manually

errors. We’ve already followed some suggestions from the Flyte community: • Set

inject-finalizer: true

• Using on-demand instances • Set

interruptible = False

and

retries = 3

Any further suggestions on how to troubleshoot this issue? Has anyone encountered something similar before and how did you resolve it? Thanks! 🙏

hmm this is odd, this is for regular workflows? and you have inject finalizer ( i see you enabled it)

Yes, we are running regular workflows, and we have the inject finalizer enabled

I think you are telling us the impact not the symptoms- can you find an example pod where this happened? Is this a map task

We already found the issue. The workflows were being terminated by Karpenter. In case someone else encounters this issue, we just had to add to the task

annotations={"<http://karpenter.sh/do-not-disrupt|karpenter.sh/do-not-disrupt>": "true"},

using

PodTemplate

Thanks for your help!

Cc @faint-activity-87590