#780 feat: add NonRecoverableError to skip retries on terminal failures
Pull request opened by
AdilFayyaz
Motivation
Tasks with retries would always exhaust all attempts even for failures that are definitively terminal (e.g. invalid input, permanent downstream unavailability). Users had no way to signal "do not retry this."
Summary
• Add NonRecoverableError(RuntimeUserError) to flyte.errors — raise it inside a task to skip all remaining
retries
• Extend the Error dataclass with a recoverable: bool = True field; convert_from_native_to_error() sets it False for NonRecoverableError
• upload_error() now maps recoverable=False → ContainerError.NON_RECOVERABLE, telling the engine to skip
retries
• Local controller retry loop breaks early when err.recoverable is False; _bin/runtime.py safety-net handler extended to also catch RuntimeUserError
Test Plan
• uv run pytest tests/flyte/internal/bin/test_runtime.py -v — includes new
test_non_recoverable_error_sets_kind asserting ContainerError.NON_RECOVERABLE is set
• Run example: uv run python examples/basics/non_recoverable_error.py
• Confirm on the UI how many retries take place with a NonRecoverableError error type.
flyteorg/flyte-sdk