https://flyte.org
Join Slack
A relatively straight forward question: how can we add a timeout on a task and let a corresponding m...
# flyte-support
b
A relatively straight forward question: how can we add a timeout on a task and let a corresponding map_task succeed? 🧵
🐛 1
What I have tried:
Copy code
@task(timeout=10)
def task1(a: int):
    time.sleep(a)

@workflow
def my_workflow():
    result = map_task(task1, min_success_ratio=0.0)(a=[1,2,10,11,12])
However the map_task fails with a timeout as 
[1/1] currentAttempt done. Last Error: USER::task execution timeout [10s] expired
b
Hi Len, what behavior are you expecting?
b
Great question. I would like the set of tasks to complete even when an input might trigger compute that lasts longer than the timeout allowed.
And specifically, I would expect result to be of length 2
I am not sure if that most recent expectation ^ is reasonable or not.
b
so timeout is the maximum amount of time a task can run before timing out… is there a reason you don’t want to increase the timeout?
or do you want the map task to pass through failing tasks and return the succeeded tasks?
b
I would like it to only return succeeded tasks.
b
you can specify 
min_success_ratio
in the map_task definition to say what percentage of tasks within the maptask would count as an overall success
b
My particular use case: • I have three tasks (each of which I am profiling different nearest neighbors search frameworks) • I have a product of inputs, some of which will cause certain tasks to take forever, but other tasks will finish in a reasonable amount of time (gpu enabled)
@broad-monitor-993 I have tried that as shown in my snippet above, but the overall map_task fails.
b
hmm, this seems like a bug @high-accountant-32689 @thankful-minister-83577 @glamorous-carpet-83516
b
OK, I’ll try to make an actual mre. Note that I tried timeout locally and it seems ignored.
b
unless maptask 
min_success_ratio
only considers exceptions raised within the actual task and not timed out tasks…
b
That is my guess tbh
b
are you running these locally btw?
b
I started locally, but timeout didn’t seem to do anything so I started developing remotely and fast registering.
That is where I saw the first error that the map_task failed
b
okay, just making sure you’re seeing this on an actual Flyte cluster
👍 1
h
@bored-beard-89967, mind opening a github issue? This is bug in the current implementation of map tasks.
b
Thanks all! I appreciate the attention on this. 🙏
i
@elegant-toddler-67101
65 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by