<#7289 fix(executor): bound consecutive Plugin.Han...
# flyte-github
a
#7289 fix(executor): bound consecutive Plugin.Handle errors with a system-failure counter Pull request opened by pingsutw Why are the changes needed? When the k8s API rejects pod creation via an admission webhook (e.g.
flyte-pod-webhook
denies the request because a referenced secret cannot be found in any configured secret manager),
launchResource
in
executor/pkg/plugin/k8s/plugin_manager.go
returned the error as a raw Go error with
pluginsCore.UnknownTransition
and no phase transition. In
taskaction_controller.go
, when
Plugin.Handle
returns a Go
error
, the controller only logs
"Plugin Handle failed"
, emits a
FailedPluginHandle
event, and requeues with the default duration. It does not record a phase, increment any system-failure counter, or convert to
PermanentFailure
. As a result, the TaskAction stays in
Queued
forever — the webhook will keep denying the same request, and there is no max-system-failure handling on this path. Repro: @env.task def fn(x: int) -> int: print("my secret is: ", os.getenv("TEST1")) # secret "test1" does not exist ... Before the fix, the TaskAction loops forever with the webhook denial message in the controller log:
Copy code
admission webhook "flyte-pod-webhook.flyte.org" denied the request:
  none of the secret managers injected secret [key:"test1" ...]
What changes were proposed in this pull request? In
launchResource
, before the generic system-error fall-through: 1. Detect admission-webhook denials via a small helper
isAdmissionWebhookDenial(err)
that matches the canonical error string (
"admission webhook"
+
"denied the request"
). 2. Also detect
k8serrors.IsInvalid(err)
(invalid pod specs are equally non-transient). 3. Return
pluginsCore.PhaseInfoFailure("AdmissionDenied", err.Error(), nil)
— a permanent failure phase — so the TaskAction terminates immediately with the underlying webhook message preserved for the user. How was this patch tested? Screenshot 2026-04-27 at 11 56 45 AM Manual end-to-end test using the bundled devbox: 1. Rebuilt
flyte-devbox:latest
with this change (
make -C docker/devbox-bundled build
) and restarted the devbox. 2. Ran a task that references a non-existent secret (
secrets=["test1"]
). 3. Confirmed the TaskAction transitions to
PermanentFailure
in ~8s (single attempt) instead of getting stuck:
Copy code
NAMESPACE                 NAME                      RUN                    TASKTYPE   STATUS             AGE
flytesnacks-development   rf5v6h9d79n2ndnnqkpr-a0   rf5v6h9d79n2ndnnqkpr   python     PermanentFailure   8s
`Error State`:
Copy code
Code:    AdmissionDenied
Kind:    USER
Message: admission webhook "flyte-pod-webhook.flyte.org" denied the request: ...
Labels • fixed Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. •
main
• #6583 • fix(executor): bound consecutive Plugin.Handle errors with a system-failure counter 👈 flyteorg/flyte