Does running the flyte control plane on spot instances raise any concerns? Is anyone running their control plane on spot instances and has this been a source of any difficulties?

Performance

Ok thanks.

We are, but haven't had many issues. I've been definitely considering moving the control plane to not use spot instances and only have the workloads on spot; I definitely think that the control plane in the long term should not be on spot instances, but trying to save some $$ 😅

It’s probably also a little more code to use intratask checkpoints in the case e.g. long-running model training. No one asked, but here’s a more real-world ML example of using this feature: link 🙃