New RunLLM response generated: >>>*# Question* Hey community, I am facing an issue, which is the following: RuntimeExecutionError: failed during plugin execution, caused by: output file @[s3://flyte/metadata/propeller/bap-development-apmzcrx7jtppgfdx5qfk/n7/data/0/n0/0/outputs.pb] is too large [142225647] bytes, max allowed [10485760] bytes ,where my expected output type is

List[AUResultsWrapperPerSegment]

It seems like the "descriptor" protobuf file has become too large, which I don't understand why. I have checked and I have a 29 long list with my custom dataclass as output:

@dataclass_json
@dataclass
class AUResultsWrapperPerSegment:
    segment_id: int
    preds: Dict[str, List[float]]

Can you tell me what to do otherwise? I don't want to change the limit since the max allowed [10485760] bytes is already large. # Answer To handle large outputs without changing the size limit, consider offloading the data to a storage service and passing references instead of the data itself. Flyte supports offloading complex data types like files and directories using

FlyteFile

or

FlyteDirectory

. You can refactor your task to save the data to a file and return a

FlyteFile

reference. Here's an example of how you can modify your task: 1. Write the output to a file: 2. Return a

FlyteFile

reference:

from flytekit import task, FlyteFile
import json
import os

@task
def process_data() -> FlyteFile:
    # Your data processing logic
    results = [AUResultsWrapperPerSegment(segment_id=i, preds={"example": [0.1, 0.2]}) for i in range(29)]
    
    # Write results to a file
    output_path = "/tmp/results.json"
    with open(output_path, "w") as f:
        json.dump([result.to_dict() for result in results], f)
    
    # Return a FlyteFile reference
    return FlyteFile(output_path)

This approach ensures that the large data is stored in a file and only a reference to the file is passed around, avoiding the protobuf size limit issue. Would you like more details on this approach? Sources: - Flyte Documentation on Handling Large Data - Flyte Community Discussion on Increasing Output Size Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1718618450.554279 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.