Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hey all, question about setting task pod specs to prefer `Spot` over `OnDemand` instances. We have some workflows that trigger 300+ tasks that we ideally would have run all on `Spot` instances. This works some of the time because of our general Kubernetes setup, however there are other times where instead of wait for Spot instances to become available they run on OnDemand which costs us a lot of extra $$$.

So my question is has anyone found a good way to set a strong preference or hard requirement through the `pod_spec` (using `V1PodSpec` from kubernetes python API) arg in the `@task` annotation to get tasks to run almost exclusively on Spot instances? Some help here would be greatly appreciated.

<@UNZB4NW3S> long time no chat, any ideas here?

Create a separate node pool and use label selectors

Okay, that is pretty in line with what I have been reading online. Let me get back to our infra team with some questions.

Thanks <@UNZB4NW3S>, seems like we only have a single node pool/group at the moment in EKS. Seeing if we want to create a new one for Spot only or tweak the existing one.