acoustic-carpenter-78188
03/27/2026, 7:55 PMNOTIFY action_updates and NOTIFY run_updates were executed via r.db.WithContext(ctx).Exec(...), which acquires a connection from GORM's shared pool on every call. Under concurrent load (many action updates happening simultaneously), this caused ~1s latency per NOTIFY due to pool contention — the NOTIFY itself is microseconds on PG, but waiting for a free connection dominated.
Fix:
• Added two buffered channels (actionNotifyCh, runNotifyCh, capacity 256) so callers (notifyActionUpdate, notifyRunUpdate) do a non-blocking channel send and return immediately.
• A single goroutine (startNotifyLoop) holds a dedicated *sql.Conn and drains both channels, completely bypassing the GORM pool.
• Batching: when multiple payloads queue up (common under load), they are drained and combined into a single SELECT pg_notify(...), pg_notify(...) round-trip, making it more efficient under high concurrency.
• Auto-reconnects on connection errors.
Tracking Issue
NA
Follow-up issue
NA
• main
• #6583
• Use dedicated connection for NOTIFY to avoid pool contention 👈
flyteorg/flyteacoustic-carpenter-78188
03/28/2026, 5:39 AM