Hello all, I'm having a problem with Flyte on a Kubernetes cluster... Some workflows, when launched, keep stuck on first task "Initializing" status. Flytepropeller logs gives me this:

{
  "json": {
    "exec_id": "a67t5ktm7f6fhksmtcfr",
    "ns": "machine-learning",
    "routine": "worker-32"
  },
  "level": "warning",
  "msg": "Workflow namespace[machine-learning]/name[a67t5ktm7f6fhksmtcfr] has already been terminated.",
  "ts": "2023-03-16T18:10:39Z"
}

Can anyone help me with this? I'll be much appreciated

Hey @hallowed-mouse-14616! Any idea what might be causing this issue?

@dry-ability-69144 do you have a larger log dump? This log indicates that Flyte is attempted to process a workflow CR that has already been completed - this can happen because k8s is an eventually consistent system.

I've found the problem! It was the GCS service account secret in kubernetes cluster that was missing. For some reason, it was giving me this error... Sorry the delay