#6681 Implement async event recording to unblock reconciliation workers Pull request opened by RRap0so ## Tracking issue Closes #6682 ## Why are the changes needed? Root Cause: Cross-region traffic between FlyteAdmin and its database adds significant latency (300ms-1s) to each event write. Since event recording happened synchronously in the reconciliation loop (admin_eventsink.go:93-150), each reconciliation worker thread blocks waiting for the database transaction to complete across regions. When FlyteAdmin experiences latency (300ms-1s per event), synchronous event recording blocks reconciliation workers. With hundreds or thousands of workflows, this causes: • 10x slowdown in reconciliation times • Cascading delays as workers wait for event responses • Potential DDOS of FlyteAdmin under high load • Degraded overall system performance ## What changes were proposed in this pull request? Implemented asynchronous event recording to decouple event sending from reconciliation, eliminating cross-region latency from the critical path: Key changes: • Events go into a buffered channel (1000 capacity by default), processed by a background worker • Sink() returns in ~1ms instead of blocking for cross-region latency • Rate limiting on the worker side to avoid overwhelming admin/database • In-flight tracking with sync.Map to prevent duplicate queuing • Graceful shutdown drains the queue • Added prometheus metrics: events_queued, events_processed, events_failed, queue_depth, processing_latency_ms Config addition: • EventQueueSize parameter (default: 1000) ## How was this patch tested? Added comprehensive tests for async processing, rate limiting, backpressure, graceful shutdown, network failures, retries, and high volume scenarios. Also tested in a staging server triggering 1000 executions at the time. ### Labels Please add one or more of the following labels to categorize your PR: • changed: For changes in existing functionality. • fixed: For any bug fixed. This is important to improve the readability of release notes. ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link https://bito.ai#summarystart| ### Summary by Bito This pull request implements asynchronous event recording in the AdminEventSink to enhance the performance of reconciliation workers by decoupling event sending from the reconciliation process. Key features include a buffered channel for event processing, rate limiting, in-flight tracking to prevent duplicates, and new Prometheus metrics for monitoring. Comprehensive tests have been added to ensure reliability under high load conditions. flyteorg/flyte