Another question: Is there any way to (mostly) ensure or enforce a FIFO order of execution of workflows. That older workflows runs are given scheduling priority for scheduling vs newer ones?

So when you submit a workflow for execution, propeller captures the workflow ID and adds it to the WorkQueue which is FIFO. The executor handles retries too. The order of execution may change due to multiple circumstances like failures or bottlenecks in persisting state on K8s. Some docs sections that can be helpful https://docs.flyte.org/en/latest/user_guide/concepts/component_architecture/flytepropeller_architecture.html#workqueue-workerpool https://docs.flyte.org/en/latest/deployment/configuration/performance.html#record-execution-status