ancient-wolf-19325
08/23/2024, 10:46 AM"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: output file @[<>/n0/data/0/outputs.pb] is too large [19664776] bytes, max allowed [10485760] bytes]. Error Type[*errors.NodeErrorWithCause]"
But the workflow is stuck on running on flyte console…..
I can see this issue to be similar but it doesn’t seem that this has been resolved.
https://github.com/flyteorg/flyte/issues/381
# Answer
The issue you're experiencing is indeed similar to the one described in Flyte Issue #381, where large outputs cause the workflow to get stuck instead of failing instantly. This issue has been acknowledged but not fully resolved.
To address this, you can try increasing the max-output-size-bytes configuration in the FlytePropeller config. According to this discussion, you can modify the flyte-propeller-config ConfigMap to increase the max-output-size-bytes value and then restart the FlytePropeller pod.
Would you like more detailed steps on how to modify the configuration?
Sources:
- Flyte Issue #381
- Flyte Discussion on Increasing Output Size
Slack Link
https://flyte-org.slack.com/archives/CP2HDHKE1/p1724409985.371659 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.