New RunLLM response generated: >>>*# Question* Hey folks. We've tried a couple times to upgrade Propeller to 1.13.0 and our pods restart every x minutes with

Lost leader state. Shutting down.

I'm not entirely sure how the leader election works within Propeller, any chance someone could give me some pointers to investigate further in the code? # Answer Leader election in FlytePropeller ensures that only one instance of FlytePropeller is active at a time to avoid conflicts, such as creating duplicate Pods or updating the FlyteWorkflow CRD simultaneously. This mechanism is crucial for maintaining consistency and preventing issues in task execution. According to Flyte's community discussion, leader election is used to keep a hot duplicate propeller environment. If one instance fails, Kubernetes leader election allows another instance to take over quickly. The default deployment charts typically set replicas to 2 and enable leader election, ensuring only one instance is active at a time. In the FlytePropeller code, the leader election mechanism is configured in the

manager.go

file. The

Manager

struct initializes the leader elector if the leader election is enabled. The relevant code can be found in the FlytePropeller manager:

func (m *Manager) Run(ctx context.Context) error {
    if m.leaderElector != nil {
        logger.Infof(ctx, "running with leader election")
        m.leaderElector.Run(ctx)
    } else {
        logger.Infof(ctx, "running without leader election")
        if err := m.run(ctx); err != nil {
            return err
        }
    }
    return nil
}

If you are experiencing frequent restarts due to leader election failures, it might be due to timeouts on requests to the Kubernetes API server. Adjusting the lease duration, renew deadline, and retry period can help mitigate this issue. As per another community discussion, increasing these values has helped in some cases:

lease-duration: 120s
renew-deadline: 110s
retry-period: 5s

Would you like more detailed information on how to configure these settings or further investigate the code? Sources: - Flyte Community Discussion on Leader Election - FlytePropeller Manager Code - Flyte Community Discussion on Leader Election Failures Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1722324225.577679 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.