https://flyte.org logo
Join Slack
Powered by
I have a map task with cardinality 3000. It works ...
# flyte-support
f
I have a map task with cardinality 3000. It works fine, until one of the map tasks change status. The propeller then runs into an issue recording the task event. It gives a 'Request Entity Too Large' error when it tries to record a task event:
Would appreciate if there is any suggestion as to how to address this.
f
you can increase the size of the entity
this is a grpc setting
f
Need to find where
@freezing-airport-6809 Is it a Flyte admin config? https://docs.flyte.org/en/latest/deployment/configuration/generated/flyteadmin_config.html The documentation says it is 0 (which I assume is unbounded, but I might be wrong)
In general, is it better to avoid too large map tasks and divide them into a flew smaller map tasks?
Copy code
configmap:
  adminServer:
    server:
      grpcPort:
        maxMessageSizeBytes
f
Map tasks should be smaller than 5k
We have been trying to push the size limits to higher, but 10MB works absolutely fine
But wait why is the message on Grpc so large?
It should not be, it only sending a link to the data
f
There is nothing special about the tasks.
The default is 4MB AFAIK.
f
Ya so it is sending 3k task status, some other metadata
f
With 4MB, 3K tasks means you have around 1300 bytes or so per task
f
ya i think it should be
f
I was able to run 5000 with increased GRPC limit.
🙌🏽 1
4 Views
Open in Slack
PreviousNext
Powered by