acoustic-carpenter-78188
02/22/2023, 11:03 AMn retries are allowed, n-1 of them are on spot instances and for the last attempt, Flyte resorts to a regular instance.
I tested the behaviour using this workflow ...
from flytekit import task, workflow, Resources
@task(interruptible=True, retries=5, requests=Resources(gpu="1"))
def train():
import time
time.sleep(600)
@workflow
def wf():
train()
... and by simulating a preemption with gcloud compute instances simulate-maintenance-event <node name> --zone <zone>.
Result:
• Pods with FLYTE_ATTEMPT_NUMBER from 0-4 had the spot/preemptible node selector.
• The pod with FLYTE_ATTEMPT_NUMBER=5 did not have the spot/preemptible node selector anymore.
flyteorg/flytesnacks
GitHub Actions: Mark github pre-release as Release
GitHub Actions: Publish artifacts to github release
GitHub Actions: Create Prerelease
GitHub Actions: Bump Version
✅ 26 other checks have passed
26/30 successful checksacoustic-carpenter-78188
03/02/2023, 10:02 AM