https://flyte.org logo
#ask-the-community
Title
# ask-the-community
r

Rupsha Chaudhuri

02/28/2024, 3:37 AM
Hi folks.. seeing something odd.. I have a map task. Apparently all of them have succeeded but the outer task is still runningā€¦ not quite sure why
s

Samhita Alla

02/28/2024, 10:16 AM
seems like a bug. would you mind filing an issue?
[flyte-bug]
d

Dan Rammer (hamersaw)

02/28/2024, 2:38 PM
@Rupsha Chaudhuri has this since completed? maptask has a post-processing phase where outputs from each subnode execution are validated and aggregated. For large fanout (in your case 4589) this means that there are ~5k sequential blobstore reads once all subnodes have completed, which can take a little while. In ArrayNode this I/O is parallelized to be more efficient.
k

Ketan (kumare3)

02/28/2024, 3:05 PM
Can you also please use experimental.map_task
r

Rupsha Chaudhuri

02/28/2024, 5:20 PM
I killed the last one after 4 hours.. and the rerun went through. That same task with same input data took 1 hr 45 min
@Dan Rammer (hamersaw) thanks for the explanation