acoustic-carpenter-78188
05/15/2026, 4:46 AMschedule_entities_snapshots wasn't being updated
We realized then, there was some launchplans which never got a “successfully caught up”, this launchplans were scheduled to 0 2 31 * * or to every 31st February (our user set this deliberately, expecting that the schedule to never be executed). Turned out, this was the schedule that started the memory issue
If you check this, the loop will quit once scheduledTime > now,
flyte/flyteadmin/scheduler/core/gocron_scheduler.go
Lines 224 to 239 in</flyteorg/flyte/commit/8bf1de63810474f3ed6bbc4c71caf841012383fe|8bf1de6>
| func GetCatchUpTimes(s models.SchedulableEntity, from time.Time, to time.Time) ([]time.Time, error) { |
| ----------------------------------------------------------------------------------------------------- |
| var scheduledTimes []time.Time |
| currFrom := from |
| for currFrom.Before(to) { |
| scheduledTime, err := GetScheduledTime(s, currFrom) |
| if err != nil { |
| return nil, err |
| } |
| if scheduledTime.After(to) { |
| break |
| } |
| scheduledTimes = append(scheduledTimes, scheduledTime) |
| currFrom = scheduledTime |
| } |
| return scheduledTimes, nil |
| } |
but the cron library has an interesting behavior, if the cron expression is valid, but the date is invalid, it will not return any error, and instead will handle this by returning a default time.Time{} or 0001:01:01 as the date. This resulted in this block of code to never quit as scheduledTime will always be less than now, and exhaust flytescheduler's memory
As this block never quit, the snapshoter initialization in here, was never executed because the initialization wait for the missed schedules to catch up or failing, and therefore it didn't take anymore snapshot and our flytescheduler kept trying to catch up the schedule from the same point
Expected behavior
flytescheduler will not try to catch up on schedule with an invalid cron date
Additional context to reproduce
No response
Screenshots
The log where flytescheduler tried to catchup the same schedule from the same time and got "caught up successfully on the schedule":
Image
Our flytescheduler memory:
Image
Are you sure this issue hasn't been raised already?
• Yes
Have you read the Code of Conduct?
• Yes
flyteorg/flyteacoustic-carpenter-78188
05/15/2026, 4:46 AM