Could anyone comment on the queue implementation in Flyte? Are those queues durable? What happens to the queue state in case of flyte-binary unexpected restart? Are there any settings we can change with respect to queue durability? CC @square-carpet-13590 @clever-exabyte-82294

I can't speak to the queue implementation - but one thing that helped our workflows to survive restarts was this setting

Thanks for this, @ripe-smartphone-56353

by the way, did it add any overhead time for your workflows?

I don't know.

I have checked the source code, looks like Flyte queues are backed by

workqueue

implementation, which is a non-durable in-memory queue AFAIK. Here is the starting code for queue implementation: https://github.com/flyteorg/flyte/blob/bed761c33c40af23750467c828afea553c0b80a0/flytepropeller/pkg/controller/workqueue.go#L7

@glamorous-rainbow-77959 when referring to queues IIUC your inquiring about the durability of workflow executions in the case of failures? Flyte achieves this by building FlytePropeller as a k8s operator. So when a workflow is executed we create a K8s CR (FlyteWorkflow). Using an informer cache, FlytePropeller operates over this FlyteWorkflow CR, updating the execution status as progress is made. These updates are designed to be singular and idempotent; for example, if we create a k8s Pod to execute a task we update the CR that the pod was created before moving to the next operation (ex. checking status of pod). This way, if there is a failure in storing state (or complete failure of FlytePropeller) we reattempt the operation (in this case creating the pod) and see it already exists before attempting to store state again. So in a long-winded way, answering your question about Flytes queue implementation. The internal

queue

is basically a list of FyteWorkflow CR IDs that FlytePropeller knows to process. If FlytePropeller dies and comes back up, this is repopulated from the list of non-terminal FlyteWorkflow CRs. All state for these executions is durably stored in etcd as part of k8s apiserver ops. @average-finland-92144 not sure if we have docs highlighting the durability guarantees? It may be worth including this information in a FlytePropeller series?

not sure if we have docs highlighting the durability guarantees?

I don't think this is explicitly covered in the docs

It may be worth including this information in a FlytePropeller series?

Absolutely

It is durable across restarts or even if the entire system Is down for hours

@hallowed-mouse-14616 thanks for the explanation. My usecase was a bit different: suppose that max_parallelism of a workflow is set to 10, but 100 execution requests arrive. 10 executions will start immediately, while the remaining 90 workflows will be queued up. So, would these queued workflows survive FlytePropeller restarts? Do we create CRs for workflows immediately even if they are waiting due to constraints like max_parallelism?

Yes they will survive- you will not lose any workflows