#7093 feat(executor): in-place task retry with scoped output paths per attempt
Pull request opened by
AdilFayyaz
Why are the changes needed?
When a task pod fails with a recoverable error, the executor relied on the runs service to create a new TaskAction CR for each retry. This external orchestration was broken, leaving retryable tasks stuck in a non-terminal RetryableFailure loop indefinitely.
What changes were proposed in this pull request?
• Intercepts PhaseRetryableFailure in TaskActionReconciler.Reconcile() before updating status
• If retries remain (currentAttempts < maxAttempts): calls p.Abort() to delete the failed pod, increments Status.Attempts, clears PluginState, and overrides the transition to PhaseQueued — causing the next reconcile to launch a fresh pod under the same TaskAction
• If retries are exhausted: converts PhaseRetryableFailure → PhasePermanentFailure, making the TaskAction terminal (Failed=True)
• Pod naming is naturally collision-free: buildGeneratedName encodes the attempt number (-0, -1, ...), so each restart creates a distinct pod
How was this patch tested?
• Deploy a task with retries: 2 (maxAttempts=3) that always exits non-zero
• Verify pods
[name]-0, [name]-1, [name]-2
are each created and deleted in sequence:
kubectl get pods -n [namespace] | grep [taskaction-name]
• Verify Status.Attempts increments on each failure:
kubectl get taskaction [name] -n [namespace] -o jsonpath='{.status.attempts}'
• After the third failure, verify the TaskAction is terminal:
kubectl get taskaction [name] -n [namespace] -o jsonpath='{.status.conditions}'
---
Failed=True, Reason=PermanentFailure
• Deploy a task with retries: 0 — confirm first failure immediately produces PermanentFailure
This is important to improve the readability of release notes.
Setup process
Screenshots
Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
Related PRs
Docs link
•
main
•
#6583
•
feat(executor): in-place task retry with scoped output paths per attempt 👈
flyteorg/flyte