<#6556 [BUG] Race condition leads to workflows stu...
# flyte-github
c
#6556 [BUG] Race condition leads to workflows stuck in UNDEFINED/UNKNOWN phase Issue created by Sovietaced ### Flyte & Flytekit version Flyte v1.15.3 ### Describe the bug We recently expanded our max workflow concurrency validation logic for existing running workflows to include workflows in an UNDEFINED phase and discovered that there were hundreds of workflows stuck in this state over time. After looking into the issue it appears that there is a race condition between Flyte Admin and Flyte Propeller. When one creates an execution Flyte Admin writes the CRD to the cluster and then later creates the execution in the Flyte Admin DB. It seems that between these two operations the CRD can get picked up by Flyte Propeller, evaluated in a tight loop, and fail. Workflow evaluation fails in particular because all events sent from Flyte Propeller to Flyte Admin fail with 404s since the execution has not been written to the Flyte Admin DB. The CRD will get to a point where its finalized and deleted, all before Flyte Admin writes the execution to the Flyte Admin DB. It is at this point that the execution DB model is stuck in this initial state for eternity. ### Expected behavior I would expect that Flyte Propeller wouldn't process or fail workflow CRDs until they are written to the Flyte Admin DB. ### Additional context to reproduce No response ### Screenshots No response ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte