Hi Team, What could possibly cause the workflows to go into an UNKNOWN state?

propeller logs:
---------------
{"json":{"src":"controller.go:157"},"level":"info","msg":"==\u003e Enqueueing workflow [examples-hbomax/f77eeb5cdbf1d432da25]","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-03-16T18:00:25Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-03-16T18:00:25Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-03-16T18:00:25Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-03-16T18:00:27Z"}

I had once gotten into this state because I had swapped from the flyte-binary to flyte-core deployments and didn't realize the mutatingwebhook that was installed was not removed, so ended up in a state where both were present in the cluster (because they used different names) and saw this behavior. Removing the flyte-binary copy fixed it. Even if not the case, work checking out the mutatingwebhook in case its the cause

oh okayyy. In my case, I had to redeploy flytepropeller pod each time I did a new deployment, which I am not sure why.