I'm seeing errors in workflows like this `Workflow...
# flyte-support
r
I'm seeing errors in workflows like this
Workflow[flytesnacks:dev:loadtesting.sleep_more_minutes] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n1]. EventRecordingFailed: failed to record node event, caused by: ResourceExhausted: Resource Exhausted, caused by [Admin EventSink throttling admin traffic]
Is there a setting I can tweak to increase admin event sink throughput? I don't see much load on the backing database which is the first thing that is recommended to check in the optimising performance section of the Flyte docs
t
I think this is happening on the propeller side actually not admin. This should be what you’re looking for https://github.com/flyteorg/flyte/blob/d4db50cb29fc6c1000260587720b4275f0dda765/flytepropeller/events/config.go#L27
👀 1
r
Thanks. That seemed to have helped 🙏