Anna Cunningham
10/31/2023, 6:45 PMFailed to update workflow. Error [rpc error: code = ResourceExhausted desc = trying to send message larger than max (2159575 vs. 2097152)]"
Can this max value be adjusted in helm? I'm struggling to find exactly where to change it.
I'm using FlytePropeller v1.1.47. Thanks!Ketan (kumare3)
Anna Cunningham
10/31/2023, 9:00 PMSamhita Alla
Yee
Anna Cunningham
11/01/2023, 3:38 PMDavid Espejo (he/him)
11/01/2023, 4:31 PMAnna Cunningham
11/01/2023, 4:35 PMflyte-core
helm chart.
I found an option for configuring the max message size in FlyteAdmin but I can't find a corresponding option for FlytePropeller. I can try adjusting maxLogMessageLength
in the meantime.Matthew Corley
11/01/2023, 9:16 PMAnna Cunningham
11/02/2023, 8:50 PMfrom flytekit import workflow, task
@task
def raise_giant_error() -> None:
error_line = "This is a really big error"
n_lines = 100000
raise ValueError("\n".join([error_line] * n_lines))
@workflow
def giant_error_wf() -> None:
raise_giant_error()
this results in the task being marked as FAILED showing the full giant error in the UI, but the workflow remains stuck in RUNNING and the FlytePropeller logs show Failed to update workflow. Error [rpc error: code = ResourceExhausted desc = trying to send message larger than max (2805993 vs. 2097152)]
. This is not solved by adjusting the maxMessageSizeBytes
or UseOffloadedWorkflowClosure
.
please let me know if there is more information I can provide!Ketan (kumare3)
Dan Rammer (hamersaw)
11/06/2023, 3:05 PMAnna Cunningham
11/06/2023, 4:34 PM