What are possible causes of ```RuntimeExecutionError max num

Question

What are possible causes of ```RuntimeExecutionError max number of system retry attempts 51 50 exhausted Last known status message failed at Node n2 EventRecordingFailed failed to record node event caused by EventSinkError Error sending event caused by rpc error code = DeadlineExceeded desc = context deadline exceeded ``` when running a dynamic task that iterates over a list of 35K inputs and spins up map tasks on batches of 500 inputs Checked flyte propeller logs in CloudWatch didn t see anything informative

Ketan (kumare3) · Answer

I think FlyteAdmin is timing out writing some event log

Ketan (kumare3) · Answer

this informs when a node in the workflow is completed etc

Ketan (kumare3) · Answer

It is your db falling over most likely

Ketan (kumare3) · Answer

Cc < Dan Rammer hamersaw > < Haytham Abuelfutuh> fyi