Flyte stops processing wfs. This behaviour observed when certain number of jobs submitted to flyte. Ex:- if we submit 500-700 jobs at a time then flyte works fine. when this number crosses around 1000, then it stops processing. All submitted wf will end up in running state but never to completion. A manual restart of propeller was the only fix so far given in those situations. Also observed that, though most the pods end up in completed status, flyte never removes finalizer to let those pods be terminated. Even if manual removal of finalizer not helping propeller to process the wf. added few graphs that shows status of our flyte instance. Can someone guide us on, how to stabilize it. We are running 1.13 version of it.

Hey union would be a good solution for scale and getting spas and deep help