What are possible causes of

RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[n2]. EventRecordingFailed: failed to record node event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]

when running a dynamic task that iterates over a list of 35K inputs and spins up map tasks on batches of 500 inputs? Checked flyte-propeller logs in CloudWatch, didn't see anything informative

I think FlyteAdmin is timing out writing some event log