Flyte supports building ML pipelines that can predict resource requirements (CPU, RAM, etc.) for processes by training models on historical data and using those models to estimate resources for new tasks. You can create a pipeline where you collect process metrics, train a model (e.g., regression), and then use the model to predict resources for future runs, leveraging Flyte's task environments and resource overrides to dynamically allocate resources per prediction or task execution. See
Flyte ML workloads migration guide and
Flyte resource configuration for concrete examples.
Would you like a step-by-step example or code snippet for such a pipeline?
Sources:
-
Flyte ML workloads migration guide
-
Flyte resource configuration ๐ 0 button ๐ 0 button ๐ก
Hint: Mention @herald in the thread for followups.