Hi all, has anyone ever seen a

EventSinkError

that on some workflows appears to occur occasionally, but on others happens all the time? We moved to a new k8s cluster for our flyteadmin instance (and from 3 instances to 1) and the timing of that seems to mostly line up with this error starting, but after checking the flyteadmin logs, I can see propeller sending a

CreateWorkflowEvent

and it getting successfully registered, but propeller will still throw an

EventRecordingFailed: failed to record node event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: protocol error]

@salmon-waitress-82501 it seems the network is being reset

Yep, we’re running the

lyft

branch of propeller and

Lyft-Flyte

of admin