Hi team.. I’m running into the same issue as what’s discussed here. It’s a dynamic task that is used to chunk up and trigger around 8000 map tasks… the input to the dynamic task is a list of inputs (dataclasses) for these map tasks… and each one is just 4 s3 urls.. To get around this I serialized the data classes into a single file and read that in the dynamic task to recreate the inputs to the map tasks.. but still running into this error

TooLarge: Event message exceeds maximum gRPC size limit, caused by [rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6192640 vs. 4194304)]

this isn’t in the running of the task itself it looks like. it looks like it’s the sending of the event to admin which contains the inputs. just too many s3 paths.

@little-cricket-84530 in Union we have automatic offloading - so that this problem will never happen

@thankful-minister-83577 where do you propose bumping this limit up?

I think it is

admin:
  server:
    grpc:
      maxMessageSizeBytes: <limit-in-bytes>

Running into the same issue, thanks for the solution. I think it's quite common to have long list of paths in the dynamic workflows, would it make sense to bump the limit to a higher safe value? 🙂

I didn't bump the limit up.. just got around the problem by serializing and writing the data classes into a file and giving the map task a partition number to read.

ah, that's not a bad solution either, thanks!