https://flyte.org logo
#announcements
Title
# announcements
e

Emirhan Karagül

06/01/2022, 11:13 AM
Hi everybody, We have been using flyte schedules for a while now. Yesterday something mysterious happened. A scheduled launchplan got executed 15 hours earlier. Usually the execution used to start around 3 seconds after 5:00 AM UTC. Anybody has an idea on how I can debug this and see what happened? (I hope it happened due to some cosmic bit flipping 😄) Thank you!
Copy code
~ flytectl get execution -p default -d production f38f2b53cfda08ccb000 -o yaml
closure:
  createdAt: "2022-05-31T14:03:10.025255897Z"
  duration: 501.772263113s
  outputs:
    uri: gs://<our-flyte-store>/metadata/propeller/default-production-f38f2b53cfda08ccb000/end-node/data/0/outputs.pb
  phase: SUCCEEDED
  startedAt: "2022-05-31T14:03:15.128550714Z"
  updatedAt: "2022-05-31T14:11:36.900814113Z"
  workflowId:
    domain: production
    name: flyte_workflows.collaborative_filtering.workflow.pipeline
    project: default
    resourceType: WORKFLOW
    version: 0.2.2
id:
  domain: production
  name: f38f2b53cfda08ccb000
  project: default
spec:
  launchPlan:
    domain: production
    name: hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
    project: default
    resourceType: LAUNCH_PLAN
    version: 0.2.2
  metadata:
    mode: SCHEDULED
    scheduledAt: "2022-06-01T05:00:00Z"
    systemMetadata: {}
p

Prafulla Mahindrakar

06/01/2022, 11:17 AM
Hi @Emirhan Karagül can you check if the scheduler you are running has this change https://github.com/flyteorg/flyteadmin/pull/331
e

Emirhan Karagül

06/01/2022, 11:21 AM
It was introduced in v0.6.90. How can I find out which release version it corresponds to?
p

Prafulla Mahindrakar

06/01/2022, 11:30 AM
e

Emirhan Karagül

06/01/2022, 11:31 AM
We are using a lower version, thanks for the update.
👍 1
p

Prafulla Mahindrakar

06/01/2022, 11:39 AM
One data point that might help aswell if there was restart around scheduler pods in that time frame
I hope this fix works for you but please do let us know if it doesn’t and also that you were speculating about something about bit flipping event 😅 hence I wanted to share this article https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge/amp
😄 1
e

Emirhan Karagül

06/02/2022, 12:17 PM
Haha, thanks for the article. Here's a video solely dedicated to bit flipping 😄

https://www.youtube.com/watch?v=AaZ_RSt0KP8&amp;ab_channel=Veritasium

👍 1
3 Views