#1422 [Core Feature] Support pause and unpause in scheduler in case admin is down
Issue created by
pmahindrakar-oss
Motivation: Why do you think this is important?
Scheduler uses admin for sending workflow executions at the scheduled time.
But in case of admin being down currently we have a retry mechanism which through configured number of steps using exponential backoff.
Admin being down is a special case where all scheduled workflows will go in this retry loop.
The proposal is to introduce a pause mechanism for the scheduler where all the existing schedules are paused
and resumed when admin is healthy. We will use admins health api to determine this.
When unpaused the scheduler will catch up on all the schedules till the pause time
cc :
@EngHabu @freezing-airport-6809 @evalsocket
Goal: What should the final outcome look like, ideally?
The scheduler pauses on admin down and unpauses when admin comes back up healthy and catches up on all the schedules.
Describe alternatives you've considered
The existing retry mechanism works but it unnecessarily causes all go routines looping in retries
[Optional] Propose: Link/Inline OR Additional context
If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document.
flyteorg/flyte