#1846 Fix: Allow elastic tasks to be recoverable Pull request opened by fg91 TL;DR For some failures, users might want to retry a flyte task, for other failures not. Flyte tasks can be marked as retriable by raising a

FlyteRecoverableException

. Torch's

elastic_launch

(which is used by flytekit's

Elastic

task type) always raises a

ChildFailedError

even if in the worker process e.g. a

FlyteRecoverableException

was raised. This means that Flyte's retry mechanism cannot be used for user errors in elastic tasks. Elastic tasks offer `torchrun`'s retry mechanism by specifying

max_retries

. However, this retry mechanism (which doesn't restart the pod but the worker processes within the pod) will retry all exceptions which is not always desireable. This PR makes elastic tasks work with Flyte's retry mechanism by propagating

FlyteRecoverableException

up from the worker processes. This is complicated by two timing issues/race conditions, one coming from torch's

elastic_launch

, the other one coming from Flyte. I propose to address this as best as possible with a convention. See doc strings within the code. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue NA Follow-up issue NA flyteorg/flytekit GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: lint GitHub Actions: build-integration (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: build-integration (ubuntu-latest, 3.8) GitHub Actions: Docs Warnings GitHub Actions: docs DCO: DCO ✅ 1 other check has passed 1/9 successful checks