#342 Cost Optimizations - Queuing Workloads
Issue created by
migueltol22
Motivation: Why do you think this is important?
Flyte workflows produce a set of compute tasks (both single and multi pod) that run on a set of physical machines. These physical machines are procured at different rates depending on availability at that moment (reserved, on-demand, spot, etc). At any given time a workload may be cheaper or more expensive to run on. Deferring execution at a later time (e.g when reserved instances are freed up) can provide substantial cost savings for any org adopting flyte.
Goal: What should the final outcome look like, ideally?
Reduce costs by running workflows at a later more cost efficient time for workflows in which the user has selected to opt in to the queuing.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Flyte component
• [x ] Overall
☐ Flyte Setup and Installation scripts
☐ Flyte Documentation
☐ Flyte communication (slack/email etc)
• [ x] FlytePropeller
• [ x] FlyteIDL (Flyte specification language)
• [ x] Flytekit (Python SDK)
• [ x] FlyteAdmin (Control Plane service)
• [ x] FlytePlugins
☐ DataCatalog
☐ FlyteStdlib (common libraries)
☐ FlyteConsole (UI)
☐ Other
[Optional] Propose: Link/Inline
https://docs.google.com/document/d/1nqAp5-DanklNvG6OeQ3uIacZYGvf0oXioz7CboP_CsU/edit?usp=sharing
Additional context
Add any other context or screenshots about the feature request here.
Is this a blocker for you to adopt Flyte
Please let us know if this makes it impossible to adopt Flyte
flyteorg/flyte