<#7418 feat: add worker-0 fast-fail and host maint...
# flyte-github
a
#7418 feat: add worker-0 fast-fail and host maintenance retry to clustered plugin Pull request opened by AdilFayyaz Why are the changes needed? The PR (#7417) happy-path plugin had no way to surface pod-level failures early or distinguish infrastructure evictions from user code failures, causing wasted retry budget on maintenance events and slow failure detection. What changes were proposed in this pull request? • Adds `maybeFastFailWorker0`: when
ReplicatedJobsStatus[workers].Failed > 0
, calls
flytek8s.DemystifyFailedOrPendingPod
on the rank-0 pod (
<jobset>-workers-0-0
) to surface failures before the JobSet controller sets JobSetFailed • Adds `maybeSystemRetryOnMaintenance`: on
JobSetFailed
, if
RestartOnHostMaintenance=true
, inspects the rank-0 pod and reclassifies system-caused evictions as
PhaseInfoSystemRetryableFailureWithCleanup
— retrying without charging the user's
max_restarts
budget •
GetTaskPhase
now reads
ClusteredTaskSpec
to check
FailurePolicy.RestartOnHostMaintenance
flag; both detection paths are best-effort (fall through to normal failure if the pod is already cleaned up) How was this patch tested? • Run:
go test ./flyteplugins/go/tasks/plugins/k8s/clustered/... -v
— all 15 tests pass • `TestGetTaskPhase_FastFail_NoJobsFailed`: confirms no pod lookup when
ReplicatedJobsStatus[workers].Failed == 0 → returns PhaseRunning
• `TestGetTaskPhase_MaintenanceRetry_FlagFalse`: confirms JobSetFailed returns
PhaseRetryableFailure
when RestartOnHostMaintenance=false, no pod lookup attempted • Integration coverage (actual eviction detection via
DemystifyFailedOrPendingPod
) is deferred to a later PR covering E2E tests Setup process Screenshots Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. Related PRs Stack If you do use
git town
to manage PR Stacks, the stack relevant to this PR will show below. Otherwise, you can ignore this section. •
main
• #7417 • feat: add worker-0 fast-fail and host maintenance retry to clustered plugin 👈 Docs link flyteorg/flyte