#7188 feat: resilient Watch RPC with reconnect and resourceVersion tracking
Pull request opened by
AdilFayyaz
Tracking issue
Depends on:
#7176,
#7175,
#7166
Why are the changes needed?
K8s watches time out every ~5 minutes by default. The previous implementation closed the client's stream silently on disconnect with no reconnect, making Watch unreliable for long-lived connections.
What changes were proposed in this pull request?
• Replaced the single-shot watch goroutine in AppK8sClient.Watch() with a reconnect loop (watchLoop + drainWatcher) that transparently reopens the K8s watch on unexpected closes or Error events
• Added resourceVersion tracking — extracted from every Added/Modified/Deleted/Bookmark event and passed to the next watch call, ensuring no events are missed or replayed across reconnects
• Added exponential backoff (1s → 2s → 4s → 30s max) between reconnect attempts; backoff resets on any successful event or Bookmark
• K8s Error events are now logged with code/reason/message instead of being silently dropped
How was this patch tested?
• go test ./app/internal/k8s/... -run TestWatch — 6 new tests covering:
channel close reconnect, Error event reconnect, Bookmark RV propagation to next watch call, exponential backoff timing, ctx cancel stops the goroutine, initial watch error surfaces synchronously
• go test ./app/... — full suite passes with no regressions
Setup process
Screenshots
Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
Related PRs
Docs link
•
main
•
#6583
•
feat: resilient Watch RPC with reconnect and resourceVersion tracking 👈
•
#7197
flyteorg/flyte