acoustic-carpenter-78188
03/18/2026, 5:57 PMPhaseInfoFailure return for `BadRequest`/`Invalid` K8s errors in plugin_manager.go so they are treated as permanent failures instead of being retried indefinitely
Problem
When a Kubernetes API server returns a BadRequest (400) or Invalid (422) error during resource creation (e.g., an admission webhook rejection), FlytePropeller logs the error but falls through to the generic system error handler. This causes the task to be retried indefinitely, wasting resources and delaying user feedback.
The return statement on line 264 was commented out:
} else if k8serrors.IsBadRequest(err) || k8serrors.IsInvalid(err) {
logger.Errorf(ctx, "Badly formatted resource for plugin [%s], err %s", e.id, err)
// return pluginsCore.DoTransition(pluginsCore.PhaseInfoFailure("BadTaskFormat", err.Error(), nil)), nil
}
Without the return, execution falls through to line 271 which wraps the error as a generic system error and returns UnknownTransition, causing infinite retries.
Root Cause
The sibling code in flyteplugins/go/tasks/plugins/array/k8s/subtask.go handles the same case correctly (uncommented), confirming this was an oversight.
Fix
Uncomment the return statement so `BadRequest`/`Invalid` errors immediately transition to PhasePermanentFailure with a clear "BadTaskFormat" reason. Added a unit test (jobBadRequest) that verifies the fix.
Testing
• Added TestK8sTaskExecutor_Handle_LaunchResource/jobBadRequest test that creates a fake client returning k8serrors.NewBadRequest(...) and verifies the transition phase is PhasePermanentFailure
• Existing tests remain unchanged
Closes #6531
flyteorg/flyte