Hi everyone! :wave: We’re seeing various tasks fai...
# ask-the-community
a
Hi everyone! 👋 We’re seeing various tasks fail with
terminated in the background, manually
errors. We’ve already followed some suggestions from the Flyte community: • Set
inject-finalizer: true
• Using on-demand instances • Set
interruptible = False
and
retries = 3
Any further suggestions on how to troubleshoot this issue? Has anyone encountered something similar before and how did you resolve it? Thanks! 🙏
k
hmm this is odd, this is for regular workflows? and you have inject finalizer ( i see you enabled it)
a
Yes, we are running regular workflows, and we have the inject finalizer enabled
k
I think you are telling us the impact not the symptoms- can you find an example pod where this happened? Is this a map task
a
We already found the issue. The workflows were being terminated by Karpenter. In case someone else encounters this issue, we just had to add to the task
annotations={"<http://karpenter.sh/do-not-disrupt|karpenter.sh/do-not-disrupt>": "true"},
using
PodTemplate
Thanks for your help!
k
Cc @Jan Fiedler