Has anyone seen issues with map tasks, partials, and lists of lists? It seems as if the map task is flattening lists of lists to determine the number of map inputs when used with a partial (e.g.
map(takes_a_list)(first=[[0, 1], [2, 3]])
would result in 4 map workers. Where I would expect it to result in two each of which receive a list.
07/14/2023, 10:05 PM
this is definitely not supposed to happen. can you paste in a repro?
which version of flytekit?
07/19/2023, 11:57 AM
Still on 1.5.X, working on a minimal example, but pretty sure this is the bug at this point. (We're also using partials, so that might be related? We also encountered the bug of lists given to the partial were treated as the map-list silently by the mapper, so it took us a little bit longer to detangle those two)