Hello, I'd like to use the results of a `map_task`...
# ask-the-community
s
Hello, I'd like to use the results of a
map_task
task with a
min_success_ratio
< 1.0 as input to another task. However, as the failed outputs are not filtered out of the results, flytekit is unable to convert them to inputs properly, resulting in errors like this and this. Since this happens in type engine code between tasks, even an intermediate task to filter out none values would fail. This blocks us from using
map_task
effectively in our workflows. Here is a min repro example: https://gist.github.com/sabrinalui/e5478b9557dcf370c84d5e57758b4c87 It seems
min_success_ratio
doesn't work locally but I attached the output when running with inputs
[true,true,false]
from our endpoint below. What would the lift be to (optionally) filter out failed tasks from the
map_task
output?
k
we append a nilLiteral here, so flytekit failed to deserialize it. @Haytham Abuelfutuh @Dan Rammer (hamersaw) can we remove this empty output?
d
Then there's no way of tracking which subtask failed right? I'm sure in some cases it doesn't matter, it other it might. Does flytekit support nullable values in any way?
Option
Union
? Would it be possible to declare the maptask output in a way that accepts the
nilLiteral
. Open to discussion on if this is a good idea or not as well.
k
I just test it with
Option[bool]
, it works for me.
@Sabrina Lui are you able to change your task type to optional type
s
This worked for me. Thanks so much @Kevin Su and @Dan Rammer (hamersaw) 🎉
118 Views