Our

flytepropeller

is continuously logging the following error (at a rate of ~50/second), and we have no workflows or tasks running. The last workflow was a few days ago.

{
  "json": {
    "exec_id": "f8114eedda3854878b11",
    "node": "n0/dn102/dn155",
    "ns": "dpp-default",
    "res_ver": "67801339",
    "routine": "worker-6",
    "src": "task_event_recorder.go:27",
    "wf": "dpp:default:msat.level2.workflow.level2_wf"
  },
  "level": "warning",
  "msg": "Failed to record taskEvent, error [EventAlreadyInTerminalStateError: conflicting events; destination: ABORTED, caused by [rpc error: code = FailedPrecondition desc = invalid phase change from SUCCEEDED to ABORTED for task execution {resource_type:TASK project:\"dpp\" domain:\"default\" name:\"msat.level2.proxy.run_splat\" version:\"dpp-b9ef0a90\"  node_id:\"n0-0-dn102-0-dn155\" execution_id:<project:\"dpp\" domain:\"default\" name:\"f8114eedda3854878b11\" >  0 {} [] 0}]]. Trying to record state: ABORTED. Ignoring this error!",
  "ts": "2022-08-31T16:29:14Z"
}

Is there a way to “reset” propeller and have it ignore these past errors? Seems the flyte state is in a bad state.

Hey @Justin Tyberg, very sorry about this behavior. It seems others are seeing this as well. Similarly, you can manually delete the FlyteWorkflow CRD for the execution - which will stop FlytePropeller from attempting to abort the workflow (that has already been aborted). We're working on a fix promptly.

oh. Nick already pinged about this. sorry. we’re on the same team. i should have searched first