https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
l

Len Strnad

07/18/2023, 9:32 PM
A relatively straight forward question: how can we add a timeout on a task and let a corresponding map_task succeed? 🧵
What I have tried:
Copy code
@task(timeout=10)
def task1(a: int):
    time.sleep(a)

@workflow
def my_workflow():
    result = map_task(task1, min_success_ratio=0.0)(a=[1,2,10,11,12])
However the map_task fails with a timeout as 
[1/1] currentAttempt done. Last Error: USER::task execution timeout [10s] expired
n

Niels Bantilan

07/18/2023, 9:40 PM
Hi Len, what behavior are you expecting?
l

Len Strnad

07/18/2023, 9:41 PM
Great question. I would like the set of tasks to complete even when an input might trigger compute that lasts longer than the timeout allowed.
And specifically, I would expect result to be of length 2
I am not sure if that most recent expectation ^ is reasonable or not.
n

Niels Bantilan

07/18/2023, 9:45 PM
so timeout is the maximum amount of time a task can run before timing out… is there a reason you don’t want to increase the timeout?
or do you want the map task to pass through failing tasks and return the succeeded tasks?
l

Len Strnad

07/18/2023, 9:46 PM
I would like it to only return succeeded tasks.
n

Niels Bantilan

07/18/2023, 9:47 PM
you can specify 
min_success_ratio
in the map_task definition to say what percentage of tasks within the maptask would count as an overall success
l

Len Strnad

07/18/2023, 9:47 PM
My particular use case: • I have three tasks (each of which I am profiling different nearest neighbors search frameworks) • I have a product of inputs, some of which will cause certain tasks to take forever, but other tasks will finish in a reasonable amount of time (gpu enabled)
@Niels Bantilan I have tried that as shown in my snippet above, but the overall map_task fails.
n

Niels Bantilan

07/18/2023, 9:49 PM
hmm, this seems like a bug @Eduardo Apolinario (eapolinario) @Yee @Kevin Su
l

Len Strnad

07/18/2023, 9:50 PM
OK, I’ll try to make an actual mre. Note that I tried timeout locally and it seems ignored.
n

Niels Bantilan

07/18/2023, 9:51 PM
unless maptask 
min_success_ratio
only considers exceptions raised within the actual task and not timed out tasks…
l

Len Strnad

07/18/2023, 9:51 PM
That is my guess tbh
n

Niels Bantilan

07/18/2023, 9:55 PM
are you running these locally btw?
l

Len Strnad

07/18/2023, 9:56 PM
I started locally, but timeout didn’t seem to do anything so I started developing remotely and fast registering.
That is where I saw the first error that the map_task failed
n

Niels Bantilan

07/18/2023, 9:56 PM
okay, just making sure you’re seeing this on an actual Flyte cluster
e

Eduardo Apolinario (eapolinario)

07/18/2023, 10:26 PM
@Len Strnad, mind opening a github issue? This is bug in the current implementation of map tasks.
l

Len Strnad

07/18/2023, 10:37 PM
Thanks all! I appreciate the attention on this. 🙏
n

Nizar Hattab

07/19/2023, 9:28 AM
@Ariel Kaspit
3 Views
Powered by