#6556 [BUG] Race condition leads to workflows stuck in UNDEFINED/UNKNOWN phase
Issue created by
Sovietaced
### Flyte & Flytekit version
Flyte v1.15.3
### Describe the bug
We recently expanded our max workflow concurrency validation logic for existing running workflows to include workflows in an UNDEFINED phase and discovered that there were hundreds of workflows stuck in this state over time. After looking into the issue it appears that there is a race condition between Flyte Admin and Flyte Propeller.
When one creates an execution Flyte Admin writes the CRD to the cluster and then later creates the execution in the Flyte Admin DB. It seems that between these two operations the CRD can get picked up by Flyte Propeller, evaluated in a tight loop, and fail. Workflow evaluation fails in particular because all events sent from Flyte Propeller to Flyte Admin fail with 404s since the execution has not been written to the Flyte Admin DB.
The CRD will get to a point where its finalized and deleted, all before Flyte Admin writes the execution to the Flyte Admin DB. It is at this point that the execution DB model is stuck in this initial state for eternity.
### Expected behavior
I would expect that Flyte Propeller wouldn't process or fail workflow CRDs until they are written to the Flyte Admin DB.
### Additional context to reproduce
No response
### Screenshots
No response
### Are you sure this issue hasn't been raised already?
• Yes
### Have you read the Code of Conduct?
• Yes
flyteorg/flyte