acoustic-carpenter-78188
03/18/2026, 6:34 PMCreateContainerError or CreateContainerConfigError and the configured grace period is exceeded, the error is classified as a USER permanent failure (PhaseInfoFailureWithCleanup). This prevents system retries from triggering — the task fails immediately even when retries is configured.
These errors are typically transient infrastructure issues (e.g. "failed to reserve container name", "container name already in use") that should be retried by the system.
Root Cause
In pod_helper.go, both CreateContainerError and CreateContainerConfigError use PhaseInfoFailureWithCleanup after the grace period, which creates:
• Phase: PhasePermanentFailure
• Error kind: ExecutionError_USER
USER errors do not trigger system retries (max-node-retries-system-failures), so the task fails immediately.
Fix
Changed both error paths to use PhaseInfoSystemRetryableFailureWithCleanup, which creates:
• Phase: PhaseRetryableFailure
• Error kind: ExecutionError_SYSTEM
This allows the system retry mechanism to handle transient container creation failures.
Testing
• Updated existing CreateContainerErrorOutsideGracePeriod test to assert PhaseRetryableFailure instead of PhasePermanentFailure
• Updated existing CreateContainerConfigErrorOutsideGracePeriod test similarly
• Both tests verify CleanupOnFailure() remains true
• Within-grace-period tests remain unchanged (still PhaseInitializing)
Closes #6786
flyteorg/flyte