When using map_task with a large number of inputs ...
# ask-the-community
v
When using map_task with a large number of inputs I get the following error
array size > max allowed. requested [13947]. allowed [5000].
How can I address this? Is there a way I can increase this size or is there a workaround to the size limit? Appreciate any help!
b
Hi Visak! That’s possible! You’d need to set the
maxArrayJobSize
size in the k8s-array config. The full config path should be
k8s.plugins.k8-array.maxArrayJobSize
v
Thank you! this is great
k
I recommend not to go too high
102 Views