<#6470 [BUG] invalid cron date schedule creates in...
# flyte-github
a
#6470 [BUG] invalid cron date schedule creates infinite loop in flytescheduler Issue created by bthari Describe the bug We were experiencing memory issue with our flytescheduler, as the memory kept hiking up until the pod would die because of OOMKilled. Everytime the pod was restarted, it tried to “catch up” the same schedule over and over again, even tho there was a log from the previous cycle which said, "caught up successfully on the schedule", then we found out, that the snapshot in the
schedule_entities_snapshots
wasn't being updated We realized then, there was some launchplans which never got a “successfully caught up”, this launchplans were scheduled to
0 2 31 * *
or to every 31st February (our user set this deliberately, expecting that the schedule to never be executed). Turned out, this was the schedule that started the memory issue If you check this, the loop will quit once
scheduledTime > now
, flyte/flyteadmin/scheduler/core/gocron_scheduler.go Lines 224 to 239 in</flyteorg/flyte/commit/8bf1de63810474f3ed6bbc4c71caf841012383fe|8bf1de6> | func GetCatchUpTimes(s models.SchedulableEntity, from time.Time, to time.Time) ([]time.Time, error) { | | ----------------------------------------------------------------------------------------------------- | | var scheduledTimes []time.Time | | currFrom := from | | for currFrom.Before(to) { | | scheduledTime, err := GetScheduledTime(s, currFrom) | | if err != nil { | | return nil, err | | } | | if scheduledTime.After(to) { | | break | | } | | scheduledTimes = append(scheduledTimes, scheduledTime) | | currFrom = scheduledTime | | } | | return scheduledTimes, nil | | } | but the cron library has an interesting behavior, if the cron expression is valid, but the date is invalid, it will not return any error, and instead will handle this by returning a default
time.Time{}
or
0001:01:01
as the date. This resulted in this block of code to never quit as
scheduledTime
will always be less than now, and exhaust flytescheduler's memory As this block never quit, the snapshoter initialization in here, was never executed because the initialization wait for the missed schedules to catch up or failing, and therefore it didn't take anymore snapshot and our flytescheduler kept trying to catch up the schedule from the same point Expected behavior flytescheduler will not try to catch up on schedule with an invalid cron date Additional context to reproduce No response Screenshots The log where flytescheduler tried to catchup the same schedule from the same time and got "caught up successfully on the schedule": Image Our flytescheduler memory: Image Are you sure this issue hasn't been raised already? • Yes Have you read the Code of Conduct? • Yes flyteorg/flyte