New RunLLM response generated: >>>*# Question* Hi, what is the recommended way to share "somewhat large but not huge" chunks of data between tasks in a single workflow ? In my case, I have a dict object of size about 4MB from the extract() task of an ETL pipeline. When I try to pass it on to the transform() task, I get

RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: output file @[<s3://my-s3-bucket/metadata/propeller/etl-development-f4fdbe7cf84934ed8a0b/n0/data/0/outputs.pb>] is too large [4890760] bytes, max allowed [2097152] bytes

I'm using the sandbox environment on my local machine right now. # Answer To share "somewhat large but not huge" chunks of data between tasks in Flyte, you can increase the allowed output size by configuring the

max-output-size-bytes

parameter in the FlytePropeller configuration. This parameter controls the maximum size of the output file that can be passed between tasks. According to this discussion, you can modify the

max-output-size-bytes

parameter in the FlytePropeller configuration. If you are using Helm to deploy Flyte, you can add this configuration in the

values.yaml

file. Here is an example of how to set this parameter in the

values.yaml

file:

configmap:
  core:
    propeller:
      max-output-size-bytes: 5242880  # 5MB

After updating the configuration, you need to redeploy Flyte to apply the changes. Would you like more details on how to apply this configuration or any other aspect of Flyte? Sources: - Hi there How to increase the allowed output size of the node Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1721659053.234819 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.