Hi everybody, We have been using flyte schedules for a while now. Yesterday something mysterious happened. A scheduled launchplan got executed 15 hours earlier. Usually the execution used to start around 3 seconds after 5:00 AM UTC. Anybody has an idea on how I can debug this and see what happened? (I hope it happened due to some cosmic bit flipping 😄) Thank you!

~ flytectl get execution -p default -d production f38f2b53cfda08ccb000 -o yaml
closure:
  createdAt: "2022-05-31T14:03:10.025255897Z"
  duration: 501.772263113s
  outputs:
    uri: gs://<our-flyte-store>/metadata/propeller/default-production-f38f2b53cfda08ccb000/end-node/data/0/outputs.pb
  phase: SUCCEEDED
  startedAt: "2022-05-31T14:03:15.128550714Z"
  updatedAt: "2022-05-31T14:11:36.900814113Z"
  workflowId:
    domain: production
    name: flyte_workflows.collaborative_filtering.workflow.pipeline
    project: default
    resourceType: WORKFLOW
    version: 0.2.2
id:
  domain: production
  name: f38f2b53cfda08ccb000
  project: default
spec:
  launchPlan:
    domain: production
    name: hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
    project: default
    resourceType: LAUNCH_PLAN
    version: 0.2.2
  metadata:
    mode: SCHEDULED
    scheduledAt: "2022-06-01T05:00:00Z"
    systemMetadata: {}

Hi @worried-farmer-76794 can you check if the scheduler you are running has this change https://github.com/flyteorg/flyteadmin/pull/331

It was introduced in v0.6.90. How can I find out which release version it corresponds to?

This release should contain the fix https://github.com/flyteorg/flyteadmin/pkgs/container/flytescheduler-release/16570753?tag=v0.19.3 Also there is parallel version release in flytescheduler repo https://github.com/flyteorg/flyteadmin/pkgs/container/flytescheduler/14046830?tag=v0.6.90

We are using a lower version, thanks for the update.

One data point that might help aswell if there was restart around scheduler pods in that time frame

Haha, thanks for the article. Here's a video solely dedicated to bit flipping 😄