Hey community, I am facing an issue, which is the...
# ask-the-community
Hey community, I am facing an issue, which is the following: RuntimeExecutionError: failed during plugin execution, caused by: output file @[s3://flyte/metadata/propeller/bap-development-apmzcrx7jtppgfdx5qfk/n7/data/0/n0/0/outputs.pb] is too large [142225647] bytes, max allowed [10485760] bytes ,where my expected output type is
It seems like the "descriptor" protobuf file has become too large, which I don't understand why. I have checked and I have a 29 long list with my custom dataclass as output:
Copy code
class AUResultsWrapperPerSegment:
    segment_id: int
    preds: Dict[str, List[float]]
Can you tell me what to do otherwise? I don't want to change the limit since the max allowed [10485760] bytes is already large.
You can use files For example a list of dataclass and instead of dictionary use a dataframe?
Or you can create list of files
We are working on an auto offload feature. Would love to discuss usecase
Daraframe could work, but why does it result in a large output protobuf file with a dictionary inside a dataclass?
It’s serializing all of the data to proto
Sadly proto does not support streaming so Flyte propeller has to materialize everything to memory. To prevent instability or high memory this limit is introduced, though I think you can raise it to 20MB there is a point at which it becomes too much
I understand, thanks for the help!