Hi, I was surprised that my long running interrupt...
# announcements
Hi, I was surprised that my long running interruptible training task was scheduled onto a non-spot instance after a failure (probably due to preemption). Thinking about it, it makes sense as default behavior, and is also documented:
If your task gets preempted, Flyte will retry your task on a non-spot (regular) instance. This retry will not count towards a retry that a user sets.
Still I'd like to configure Flyte to try one more time on a spot instance. There's a interruptible-failure-threshold ("number of failures for a node to be still considered interruptible"). Is this the right config to tweak that behavior?
@jeev this is the same behavior we were seeing right
@Sören Brunk yes, that is the correct configuration. It looks like here we compare the number of system failures (designation for interruptions) with the
and unset
if we see the number of failures exceed the threshold. Intuitively, you should be able to set the
to a number higher than the number of retries to ensure the task is only executed on SPOT instances.
Awesome! Thanks for the explanation @Dan Rammer (hamersaw)
👍 1