<#7188 feat: resilient Watch RPC with reconnect an...
# flyte-github
a
#7188 feat: resilient Watch RPC with reconnect and resourceVersion tracking Pull request opened by AdilFayyaz Tracking issue Depends on: #7176, #7175, #7166 Why are the changes needed? K8s watches time out every ~5 minutes by default. The previous implementation closed the client's stream silently on disconnect with no reconnect, making Watch unreliable for long-lived connections. What changes were proposed in this pull request? • Replaced the single-shot watch goroutine in AppK8sClient.Watch() with a reconnect loop (watchLoop + drainWatcher) that transparently reopens the K8s watch on unexpected closes or Error events • Added resourceVersion tracking — extracted from every Added/Modified/Deleted/Bookmark event and passed to the next watch call, ensuring no events are missed or replayed across reconnects • Added exponential backoff (1s → 2s → 4s → 30s max) between reconnect attempts; backoff resets on any successful event or Bookmark • K8s Error events are now logged with code/reason/message instead of being silently dropped How was this patch tested? • go test ./app/internal/k8s/... -run TestWatch — 6 new tests covering: channel close reconnect, Error event reconnect, Bookmark RV propagation to next watch call, exponential backoff timing, ctx cancel stops the goroutine, initial watch error surfaces synchronously • go test ./app/... — full suite passes with no regressions Setup process Screenshots Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. Related PRs Docs link •
main
• #6583 • feat: resilient Watch RPC with reconnect and resourceVersion tracking 👈 • #7197 flyteorg/flyte