We currently have four Launch Plans that are scheduled to run every day at 7:00 PM UTC with the following cron:

0 19 * * *

In most cases, the scheduling works as expected and all LPs are triggered at the correct time. However, in some cases, one or more LPs are randomly delayed by 1 or 2 hours. For example, I observed the following log entry:

msg: successfully fired the request for schedule
{
  BaseModel: {
    ID: 4869
    CreatedAt: 2026-02-27 21:26:06.546797 +0000 UTC
    UpdatedAt: 2026-02-27 21:26:06.546797 +0000 UTC
    DeletedAt: <nil>
  }
  SchedulableEntityKey: {
    Project: my-project
    Domain: my-domain
    Name: my_lp
    Version: 1.0
  }
  CronExpression: 0 19 * * *
  FixedRateValue: 0
  Unit: MINUTE
  KickoffTimeInputArg:
  Active:0xc002517c64
}
for time 2026-02-28 19:00:00 +0000 UTC

Based on the log, the expected launch time should have been

2026-02-28 19:00:00 UTC

. However, the corresponding log entry was created at

2026-02-28T20:00:03Z

, indicating a delay of one hour, which is also visible in the Flyte UI (so it was triggered with 1 hour delay without any clue). I can confirm that we currently have only one scheduler pod running. Additionally, during the time ranges when the Launch Plans were expected to be triggered, there were no observable issues with the pod (such as crashes or restarts) What is particularly strange is that the delay appears to be exactly 1 or 2 hours, rather than a random number of minutes. Has anyone encountered a similar issue before, or does anyone have ideas about what could cause this kind of scheduling delay? Thank you so much in advance.

I can confirm that this issue appears to occur randomly. For example, a LP may be triggered correctly 4-5 time in a row, and then the next scheduled run becomes delayed. During these cases, there were no changes on the Flyte backend side, and no updates of the LP itself.

For example, yesterday’s execution was correct, while the execution from the previous day had only one delayed LP.