Hi there! How to increase the allowed output size of the node?

RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [k8s-array]: output file @[] is too large [255540150] bytes, max allowed [10485760] bytes

I ran a search for

10485760

in https://github.dev/flyteorg/flyte, seems like it is the default value for

max-output-size-bytes

for flyteadmin, flytepropeller, and scheduler_config… After searching through discuss.flyte.org for mentions of

max-output-size-bytes

I found this thread where someone mentioned the exact problem you’re having: https://discuss.flyte.org/t/2714230/hi-all-wave-i-have-a-problem-of-sharing-data-between-tasks-i In their case adding

max-output-size-bytes

to the

flyte-propeller-config

configmap in the namespace where you installed flyte, followed by a restart of the flytepropeller pod, let them increase the max output size. If you installed it without helm, just edit the configmap. If you installed with helm. keep reading for how to configure this in helm: In the flyteorg/flyte github repo, there’s the values file for flyte-core, and there’s this section:

# -- Core propeller configuration
  core:
    # -- follows the structure specified [here](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config>).
    manager:
      pod-application: "flytepropeller"
      pod-template-container-name: "flytepropeller"
      pod-template-name: "flytepropeller-template"
    # -- follows the structure specified [here](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config>).
    propeller:
      rawoutput-prefix: <s3://my-s3-bucket/>
      metadata-prefix: metadata/propeller
      workers: 4
      max-workflow-retries: 30
      workflow-reeval-duration: 30s
      downstream-eval-duration: 30s
      limit-namespace: "all"
      prof-port: 10254

I checked how it’s read by the helm templates, and it is read as-is:

{{- if .Values.flytepropeller.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-propeller-config
  namespace: {{ template "flyte.namespace" . }}
  labels: {{ include "flyteadmin.labels" . | nindent 4 }}
data:
...
{{- with .Values.configmap.core }}
  core.yaml: | {{ tpl (toYaml .) $ | nindent 4 }}
{{- end }}
...

So you should be able to just add the helm value

.configmap.core.propeller.max-output-size-bytes

with the desired max output size value and it should work

Thanks for the detailed response, @Victor Churikov. Really appreciate it. If this solution proves effective, would it be possible for you to add it to our troubleshooting guide? It could be very helpful for many of our users. https://docs.flyte.org/en/latest/community/troubleshoot.html