When a task is retried in Flyte, it spins up a new Kubernetes pod for each retry attempt; pods are not reused between retries. Each task execution is isolated in its own container, and failed attempts result in new pods being created for subsequent retries, ensuring a clean environment for each run. This behavior applies to both initial attempts and all retries, and is fundamental to Flyte's execution and isolation model according to the official documentation and architecture overview (
source,
source).
If a pod fails (e.g., due to an error or resource issue), Flyte will terminate that pod and launch a new one for the next retry, rather than attempting to recover or reuse the failed pod. This approach ensures reproducibility and avoids contamination from previous failed states (
source).
Would you like more detail on how Flyte manages pod lifecycle or how this impacts resource usage?
Sources:
-
https://www.union.ai/docs/v2/flyte/user-guide/task-configuration/retries-and-timeouts/
-
https://github.com/flyteorg/flyte/blob/master/docs/user_guide/basics/tasks.md 👍 0 button 👎 0 button 💡
Hint: Mention @runllm in the thread for followups.