We tend to have some pretty large fan outs and occ...
# ask-the-communityg
Greg Gydush
03/07/2023, 4:15 PMWe tend to have some pretty large fan outs and occasionally get hit with the gRPC size limit and wondering if we could think about strategies to prevent users hitting this too. In the most recent case, I had a dynamic that has a list of 4238 items, and will process a task for each item. Each item in the list only has 2 attributes (dataset_id, raw_data_path), but it still exceeds gRPC limits.
I'm wondering if it could be possible for Flyte to automatically see that the list is getting large, and serialize to file to prevent gRPC issues (and potentially do this for all collection types)
Greg Gydush
03/07/2023, 4:20 PMIn our case, our gRPC limit is 4 mb - is this something that could be configurable? Something like 10-20mb would be much more enabling as a quick-fix
d
Dan Rammer (hamersaw)
03/07/2023, 4:32 PMHey @Greg Gydush grpc size limits on FlyteAdmin should be configurable right now. I believe this is the configuration option - ie.
maxMessageSizeBytesDan Rammer (hamersaw)
03/07/2023, 4:34 PMDuring this fix I know we discussed dynamically checking if the grpc message size was larger than a certains bounds; and if so, splitting the message into multiple and using a streaming protocol, but if i recall this added a considerable amount of overhead. I think the aforementioned configuration should alleviate your issues, but if not we can certainly revisit this!
g
Greg Gydush
03/07/2023, 4:35 PMExcellent! Do you see any issues increasing to 20mb?
d
Dan Rammer (hamersaw)
03/07/2023, 4:36 PMI know that gRPC had pretty specific reasons for having a 4MB default limit. I think it had to do with performance, but not certain I remember correctly. Maybe just keep an eye out for anything that looks irregular as far as performance is concerned.
151 Views