white-yak-77603
07/01/2023, 7:37 AMRuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [k8s-array]: output file @[] is too large [255540150] bytes, max allowed [10485760] bytes
kind-kite-58745
07/02/2023, 6:18 PM10485760
in https://github.dev/flyteorg/flyte, seems like it is the default value for max-output-size-bytes
for flyteadmin, flytepropeller, and scheduler_config…
After searching through discuss.flyte.org for mentions of max-output-size-bytes
I found this thread where someone mentioned the exact problem you’re having: https://discuss.flyte.org/t/2714230/hi-all-wave-i-have-a-problem-of-sharing-data-between-tasks-i
In their case adding max-output-size-bytes
to the flyte-propeller-config
configmap in the namespace where you installed flyte, followed by a restart of the flytepropeller pod, let them increase the max output size.
If you installed it without helm, just edit the configmap. If you installed with helm. keep reading for how to configure this in helm:
In the flyteorg/flyte github repo, there’s the values file for flyte-core, and there’s this section:
# -- Core propeller configuration
core:
# -- follows the structure specified [here](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config>).
manager:
pod-application: "flytepropeller"
pod-template-container-name: "flytepropeller"
pod-template-name: "flytepropeller-template"
# -- follows the structure specified [here](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config>).
propeller:
rawoutput-prefix: <s3://my-s3-bucket/>
metadata-prefix: metadata/propeller
workers: 4
max-workflow-retries: 30
workflow-reeval-duration: 30s
downstream-eval-duration: 30s
limit-namespace: "all"
prof-port: 10254
I checked how it’s read by the helm templates, and it is read as-is:
{{- if .Values.flytepropeller.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: flyte-propeller-config
namespace: {{ template "flyte.namespace" . }}
labels: {{ include "flyteadmin.labels" . | nindent 4 }}
data:
...
{{- with .Values.configmap.core }}
core.yaml: | {{ tpl (toYaml .) $ | nindent 4 }}
{{- end }}
...
So you should be able to just add the helm value .configmap.core.propeller.max-output-size-bytes
with the desired max output size value and it should worktall-lock-23197