New RunLLM response generated: >>>*# Question* I'm seeing errors in workflows like this

Workflow[flytesnacks:dev:loadtesting.sleep_more_minutes] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n1]. EventRecordingFailed: failed to record node event, caused by: ResourceExhausted: Resource Exhausted, caused by [Admin EventSink throttling admin traffic]

Is there a setting I can tweak to increase admin event sink throughput? I don't see much load on the backing database which is the first thing that is recommended to check in the optimising performance section of the Flyte docs # Answer To address the

ResourceExhausted

error caused by the Admin EventSink throttling admin traffic, you can adjust the rate and capacity settings for the Admin EventSink. According to the FlytePropeller events configuration, you can configure the rate and capacity parameters to control the throughput. Here is an example configuration snippet to increase the rate and capacity:

admin:
  event:
    rate: 1000  # Increase the rate limit
    capacity: 2000  # Increase the capacity

This configuration should be added to your FlytePropeller configuration file. Adjust the values as needed based on your workload and performance requirements. Would you like more details on how to apply this configuration or any other aspect of Flyte's performance tuning? Sources: - FlytePropeller events configuration Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1725432747.462379 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.