Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello, does anyone know what’s the expected behaviour when Flyte’s control plane restarts while a workflow that uses <https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/control_flow/dynamics.html#id1|dynamic workflows> and <https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/control_flow/waiting_for_external_inputs.html#waiting-for-external-inputs|waiting for external inputs> such as using the <https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/control_flow/waiting_for_external_inputs.html#pause-executions-with-the-sleep-node|sleep> node is running?

Is it able to gracefully survive the restart of the control plane and resume the traversal of the graph (for the dynamic workflow) and continue sleeping up till the expected duration (for the sleep node)?

Ya it should. Flyte propeller maintains state in etcD, so it should survive even DB restarts. Also propeller is designed stateless, with discrete steps stored and fully recoverable- so even propeller down for a bit should not impact anything

The system is designed to be resilient to outages and failures in single
Components.
Execution in progress state is lost only when kubernetes crashes, but completed state can be fully recovered