#6682 [BUG] Synchronous event recording blocks reconciliation workers under cross-region latency Issue created by RRap0so Flyte & Flytekit version: Flyte v1.15.3 ### Describe the bug Synchronous event recording in flytepropeller blocks reconciliation workers when FlyteAdmin experiences high latency, particularly in cross-region deployments where the database is in a different region. Each event recording call (CreateWorkflowEvent, CreateNodeEvent, CreateTaskEvent) blocks the reconciliation worker thread until FlyteAdmin completes the database write. With cross-region latency of 300ms-1s per event, this causes, Self-inflicted DDOS under high workflow volumes. ### Expected behavior Event recording should not block reconciliation workers. Events should be queued asynchronously so that: • Reconciliation workers can immediately continue processing workflows • Cross-region latency only affects event recording throughput, not reconciliation latency • The system remains responsive under high load ### Additional context to reproduce 1. Deploy FlyteAdmin in one region 2. Deploy FlyteAdmin's database in a different region 3. Run 100+ concurrent workflows that generate events 4. Observe reconciliation times increase from seconds to minutes 5. Check worker metrics - all threads blocked in event recording ### Screenshots Logs: Context Cancelled and Deadline bringing execution to an halt until traffic would be transferred to the same region of the database. [Image](https://private-user-images.githubusercontent.com/100569684/502279038-7d3113ba-09c7-471b-b2e6-d1dc02d56631.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjA5NjM2NDAsIm5iZiI6MTc2MDk2MzM0MCwicGF0aCI6Ii8xMDA1Njk2ODQvNTAyMjc5MDM4LTdkMzExM2JhLTA5YzctNDcxYi1iMmU2LWQxZGMwMmQ1NjYzMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUxMDIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MTAyMFQxMjI5MDBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hMTBkYzcxODViYTRlNzFhYzViMzY0OThiNDdkYjQ3NmNlYjExNzc4NGFlNTJlYTE1YmRlNmFkYjUyNzBiZWRlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.-2RdeukC0z7w-1hu2PBp9Mwve6rUZO5ywdPtVjHQHNA) [Image](https://private-user-images.githubusercontent.com/100569684/502279040-ad6ae6f4-8b45-46cf-90d9-305bd12471ba.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjA5NjM2NDAsIm5iZiI6MTc2MDk2MzM0MCwicGF0aCI6Ii8xMDA1Njk2ODQvNTAyMjc5MDQwLWFkNmFlNmY0LThiNDUtNDZjZi05MGQ5LTMwNWJkMTI0NzFiYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUxMDIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MTAyMFQxMjI5MDBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kYTgxNTcwMjBmZDlhM2FkNmVmNGE1Mzg4YTdhMTRiYmU5NTNhMDNmYWZhMjZlOGQ5MDVmOTU2ODQ0MDc3OGJhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.c0IWIv1FnX0Wl4Zam3VBw3V0jfw08WdoOTtfEnGN_V0) [Image](https://private-user-images.githubusercontent.com/100569684/502279039-a0f5939a-93a0-4b2f-bc4a-374e85e0b35c.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjA5NjM2NDAsIm5iZiI6MTc2MDk2MzM0MCwicGF0aCI6Ii8xMDA1Njk2ODQvNTAyMjc5MDM5LWEwZjU5MzlhLTkzYTAtNGIyZi1iYzRhLTM3NGU4NWUwYjM1Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUxMDIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MTAyMFQxMjI5MDBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yYzNjNDg3YmQwMzQ2NDI1M2I0MDVlYTE1MzI4ODMxZTYxOTUwODhjODMwYjYzODViZTY2ZjAyMjBhYmZjOGJmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.F1FuCdSyrjrgwaROv9JzS92Rp6Sacf3dQatQc1iuBVU) ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte