Justin Tyberg
08/31/2022, 4:35 PMflytepropeller
is continuously logging the following error (at a rate of ~50/second), and we have no workflows or tasks running. The last workflow was a few days ago.
{
"json": {
"exec_id": "f8114eedda3854878b11",
"node": "n0/dn102/dn155",
"ns": "dpp-default",
"res_ver": "67801339",
"routine": "worker-6",
"src": "task_event_recorder.go:27",
"wf": "dpp:default:msat.level2.workflow.level2_wf"
},
"level": "warning",
"msg": "Failed to record taskEvent, error [EventAlreadyInTerminalStateError: conflicting events; destination: ABORTED, caused by [rpc error: code = FailedPrecondition desc = invalid phase change from SUCCEEDED to ABORTED for task execution {resource_type:TASK project:\"dpp\" domain:\"default\" name:\"msat.level2.proxy.run_splat\" version:\"dpp-b9ef0a90\" node_id:\"n0-0-dn102-0-dn155\" execution_id:<project:\"dpp\" domain:\"default\" name:\"f8114eedda3854878b11\" > 0 {} [] 0}]]. Trying to record state: ABORTED. Ignoring this error!",
"ts": "2022-08-31T16:29:14Z"
}
Is there a way to “reset” propeller and have it ignore these past errors? Seems the flyte state is in a bad state.Dan Rammer (hamersaw)
08/31/2022, 4:40 PMJustin Tyberg
08/31/2022, 4:41 PM