When using map_task with a large number of inputs I get the following error

array size > max allowed. requested [13947]. allowed [5000].

How can I address this? Is there a way I can increase this size or is there a workaround to the size limit? Appreciate any help!

Hi Visak! That’s possible! You’d need to set the

maxArrayJobSize

size in the k8s-array config. The full config path should be

k8s.plugins.k8-array.maxArrayJobSize

Thank you! this is great

I recommend not to go too high