How do retries and timeouts interact with each oth...
# flyte-supportg
gentle-tomato-480
04/02/2025, 8:29 AMHow do retries and timeouts interact with each other?
I have a task for which I've set a 5 minute timeout and 3 retries.
In the most recent runs from last night, I see that due to lack of resources (it requires a gpu, but we scale our gpu pool of nodes down to 0 overnight) it times out after 10 minutes and only got retried twice. I would expect it to run at most 15 minutes (3x5min) instead of 10
d
damp-lion-88352
04/02/2025, 8:31 AMI don't think we will retry if there's a timeout error
damp-lion-88352
04/02/2025, 8:31 AMcan you provide your code?
g
gentle-tomato-480
04/02/2025, 8:32 AMCan I just give the @task decorator? As the actual task doesn't get run, and when it does run, it normally doesn't time out.
d
damp-lion-88352
04/02/2025, 8:32 AMsrue
damp-lion-88352
04/02/2025, 8:32 AMsure
damp-lion-88352
04/02/2025, 8:32 AMin my experiece, if you reach timeout error, we will not retry
g
gentle-tomato-480
04/02/2025, 8:32 AM Copy code
@task(container_image=image_spec, requests=Resources(cpu="1", mem="2G"), accelerator=GPUAccelerator("nvidia-l4"), limits=Resources(cpu="4", mem="7G", gpu="1"), timeout=timedelta(minutes=5), retries=3)gentle-tomato-480
04/02/2025, 8:32 AMOkay, is that because the timeout is a system error?
d
damp-lion-88352
04/02/2025, 8:33 AMyes I think is
damp-lion-88352
04/02/2025, 8:33 AMmaybe you need to provide your console's screenshot and more log from propeller
g
gentle-tomato-480
04/02/2025, 8:33 AMOkay
d
damp-lion-88352
04/02/2025, 8:34 AMso that it will be more easier to help use figure out what happened
damp-lion-88352
04/02/2025, 8:34 AMtyty
g
gentle-tomato-480
04/02/2025, 8:39 AMThe task at hand is the
download_rawd
damp-lion-88352
04/02/2025, 8:40 AMbusy on other priority
damp-lion-88352
04/02/2025, 8:40 AMwill come back if I have time today
damp-lion-88352
04/02/2025, 8:40 AMsry
g
gentle-tomato-480
04/02/2025, 8:40 AMNo worries 👍 Good luck
d
damp-lion-88352
04/02/2025, 8:40 AMand the file is not found
damp-lion-88352
04/02/2025, 8:41 AMI can
damp-lion-88352
04/02/2025, 8:41 AMcan't open it
g
gentle-tomato-480
04/02/2025, 8:41 AMYeah, forgot to edit one more sensitive thing out. One sec
gentle-tomato-480
04/02/2025, 8:43 AMHere you go. It's not high priority, just curious about if this is expected behaviour and the relationship between timeouts and retries so I can tune my workflows better.
gentle-tomato-480
04/02/2025, 8:47 AMhttps://flyte--6382.org.readthedocs.build/en/6382/user_guide/flyte_fundamentals/optimizing_tasks.html#configuring-retries Ah, if I read correctly. It might be that somewhere in my flytepropeller config the default number of system error based retries is 1 or 2. (Is attempts == retries or attempts = 1 + retries?)
gentle-tomato-480
04/02/2025, 8:51 AMhttps://github.com/flyteorg/flyte/blob/64d053476affd3db7e580fe05ce850f1476f6ae3/flytepropeller/pkg/controller/config/config.go#L280 Seems to be the case if I read it correctly, yeah. Cool
c
careful-australia-19356
04/09/2025, 4:42 PM@gentle-tomato-480 from that line it seems like it's attempts = 1 + retries. Have you been able to test this?
g
gentle-tomato-480
04/09/2025, 4:43 PMFor our needs, 2 turned out to be enough. So I haven't touched/changed it in config
5 Views