A relatively straight forward question: how can we...
# ask-the-communityl
Len Strnad
07/18/2023, 9:32 PMA relatively straight forward question: how can we add a timeout on a task and let a corresponding map_task succeed? 🧵
Len Strnad
07/18/2023, 9:34 PMWhat I have tried:
Copy code
@task(timeout=10)
def task1(a: int):
time.sleep(a)
@workflow
def my_workflow():
result = map_task(task1, min_success_ratio=0.0)(a=[1,2,10,11,12])Len Strnad
07/18/2023, 9:34 PMHowever the map_task fails with a timeout as
[1/1] currentAttempt done. Last Error: USER::task execution timeout [10s] expiredn
Niels Bantilan
07/18/2023, 9:40 PMHi Len, what behavior are you expecting?
l
Len Strnad
07/18/2023, 9:41 PMGreat question. I would like the set of tasks to complete even when an input might trigger compute that lasts longer than the timeout allowed.
Len Strnad
07/18/2023, 9:42 PMAnd specifically, I would expect result to be of length 2
Len Strnad
07/18/2023, 9:42 PMI am not sure if that most recent expectation ^ is reasonable or not.
n
Niels Bantilan
07/18/2023, 9:45 PMso timeout is the maximum amount of time a task can run before timing out… is there a reason you don’t want to increase the timeout?
Niels Bantilan
07/18/2023, 9:46 PMor do you want the map task to pass through failing tasks and return the succeeded tasks?
l
Len Strnad
07/18/2023, 9:46 PMI would like it to only return succeeded tasks.
n
Niels Bantilan
07/18/2023, 9:47 PMyou can specify
min_success_ratiol
Len Strnad
07/18/2023, 9:47 PMMy particular use case:
• I have three tasks (each of which I am profiling different nearest neighbors search frameworks)
• I have a product of inputs, some of which will cause certain tasks to take forever, but other tasks will finish in a reasonable amount of time (gpu enabled)
Len Strnad
07/18/2023, 9:49 PM@Niels Bantilan I have tried that as shown in my snippet above, but the overall map_task fails.
n
Niels Bantilan
07/18/2023, 9:49 PMhmm, this seems like a bug @Eduardo Apolinario (eapolinario) @Yee @Kevin Su
l
Len Strnad
07/18/2023, 9:50 PMOK, I’ll try to make an actual mre. Note that I tried timeout locally and it seems ignored.
n
Niels Bantilan
07/18/2023, 9:51 PMunless maptask
min_success_ratiol
Len Strnad
07/18/2023, 9:51 PMThat is my guess tbh
n
Niels Bantilan
07/18/2023, 9:55 PMare you running these locally btw?
l
Len Strnad
07/18/2023, 9:56 PMI started locally, but timeout didn’t seem to do anything so I started developing remotely and fast registering.
Len Strnad
07/18/2023, 9:56 PMThat is where I saw the first error that the map_task failed
n
Niels Bantilan
07/18/2023, 9:56 PMokay, just making sure you’re seeing this on an actual Flyte cluster
e
Eduardo Apolinario (eapolinario)
07/18/2023, 10:26 PM@Len Strnad, mind opening a github issue? This is bug in the current implementation of map tasks.
l
Len Strnad
07/18/2023, 10:37 PMLen Strnad
07/18/2023, 10:39 PMThanks all! I appreciate the attention on this. 🙏
n
Nizar Hattab
07/19/2023, 9:28 AM@Ariel Kaspit
25 Views