Thread
#announcements
    Emirhan Karagül

    Emirhan Karagül

    3 months ago
    Hi everybody, We have been using flyte schedules for a while now. Yesterday something mysterious happened. A scheduled launchplan got executed 15 hours earlier. Usually the execution used to start around 3 seconds after 5:00 AM UTC. Anybody has an idea on how I can debug this and see what happened? (I hope it happened due to some cosmic bit flipping 😄) Thank you!
    ~ flytectl get execution -p default -d production f38f2b53cfda08ccb000 -o yaml
    closure:
      createdAt: "2022-05-31T14:03:10.025255897Z"
      duration: 501.772263113s
      outputs:
        uri: gs://<our-flyte-store>/metadata/propeller/default-production-f38f2b53cfda08ccb000/end-node/data/0/outputs.pb
      phase: SUCCEEDED
      startedAt: "2022-05-31T14:03:15.128550714Z"
      updatedAt: "2022-05-31T14:11:36.900814113Z"
      workflowId:
        domain: production
        name: flyte_workflows.collaborative_filtering.workflow.pipeline
        project: default
        resourceType: WORKFLOW
        version: 0.2.2
    id:
      domain: production
      name: f38f2b53cfda08ccb000
      project: default
    spec:
      launchPlan:
        domain: production
        name: hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
        project: default
        resourceType: LAUNCH_PLAN
        version: 0.2.2
      metadata:
        mode: SCHEDULED
        scheduledAt: "2022-06-01T05:00:00Z"
        systemMetadata: {}
    p

    Prafulla Mahindrakar

    3 months ago
    Hi @Emirhan Karagül can you check if the scheduler you are running has this change https://github.com/flyteorg/flyteadmin/pull/331
    Emirhan Karagül

    Emirhan Karagül

    3 months ago
    It was introduced in v0.6.90. How can I find out which release version it corresponds to?
    p

    Prafulla Mahindrakar

    3 months ago
    Emirhan Karagül

    Emirhan Karagül

    3 months ago
    We are using a lower version, thanks for the update.
    p

    Prafulla Mahindrakar

    3 months ago
    One data point that might help aswell if there was restart around scheduler pods in that time frame
    I hope this fix works for you but please do let us know if it doesn’t and also that you were speculating about something about bit flipping event 😅 hence I wanted to share this article https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge/amp
    Emirhan Karagül

    Emirhan Karagül

    3 months ago
    Haha, thanks for the article. Here's a video solely dedicated to bit flipping 😄

    https://www.youtube.com/watch?v=AaZ_RSt0KP8&amp;ab_channel=Veritasium