Hey all, question about setting task pod specs to prefer

Spot

over

OnDemand

instances. We have some workflows that trigger 300+ tasks that we ideally would have run all on

Spot

instances. This works some of the time because of our general Kubernetes setup, however there are other times where instead of wait for Spot instances to become available they run on OnDemand which costs us a lot of extra $$$. So my question is has anyone found a good way to set a strong preference or hard requirement through the

pod_spec

(using

V1PodSpec

from kubernetes python API) arg in the

@task

annotation to get tasks to run almost exclusively on Spot instances? Some help here would be greatly appreciated.

Create a separate node pool and use label selectors

Okay, that is pretty in line with what I have been reading online. Let me get back to our infra team with some questions.