https://flyte.org
Join Slack
We tend to have some pretty large fan outs and occasionally get hit with the gRPC size limit and won...
# flyte-support
r
We tend to have some pretty large fan outs and occasionally get hit with the gRPC size limit and wondering if we could think about strategies to prevent users hitting this too. In the most recent case, I had a dynamic that has a list of 4238 items, and will process a task for each item. Each item in the list only has 2 attributes (dataset_id, raw_data_path), but it still exceeds gRPC limits. I'm wondering if it could be possible for Flyte to automatically see that the list is getting large, and serialize to file to prevent gRPC issues (and potentially do this for all collection types)
In our case, our gRPC limit is 4 mb - is this something that could be configurable? Something like 10-20mb would be much more enabling as a quick-fix
h
Hey @rich-garden-69988 grpc size limits on FlyteAdmin should be configurable right now. I believe this is the configuration option - ie. 
maxMessageSizeBytes
.
❤️ 1
During this fix I know we discussed dynamically checking if the grpc message size was larger than a certains bounds; and if so, splitting the message into multiple and using a streaming protocol, but if i recall this added a considerable amount of overhead. I think the aforementioned configuration should alleviate your issues, but if not we can certainly revisit this!
👍 1
r
Excellent! Do you see any issues increasing to 20mb?
h
I know that gRPC had pretty specific reasons for having a 4MB default limit. I think it had to do with performance, but not certain I remember correctly. Maybe just keep an eye out for anything that looks irregular as far as performance is concerned.
👍 1
165 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by