calm-pilot-2010
11/10/2023, 12:52 PME1106 17:54:54.406295 1 leaderelection.go:369] Failed to update lock: Put "<https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/infrastructure--helm--flyte/leases/propeller-leader?timeout=2m0s>": context deadline exceeded
I1106 17:54:54.406322 1 leaderelection.go:285] failed to renew lease infrastructure--helm--flyte/propeller-leader: timed out waiting for the condition
{"json":{},"level":"fatal","msg":"Lost leader state. Shutting down.","ts":"2023-11-06T17:54:54Z"}
Given that we are not currently using a shared flyte-propeller, I thinking we may be able to disable leader election entirely?hallowed-mouse-14616
11/10/2023, 1:01 PMcalm-pilot-2010
11/10/2023, 1:04 PMhallowed-mouse-14616
11/10/2023, 1:39 PMthen I should anyway never have more than one flyte propeller running simultaneously?Correct!
How would the duplicate propeller env you describe be configured?I know the default deployment charts for a long time just set
replicas: 2
and enabled leader election. Then k8s starts 2 propeller instances automatically and the leader-election mechanism ensures only one is active at a time.hallowed-mouse-14616
11/10/2023, 1:41 PMcalm-pilot-2010
11/10/2023, 1:57 PMflyte-core
helm. Probably it would be a trivial helm PR, but I think I will just make the leases and renew periods a bit longer to mitigate our problem.freezing-airport-6809
white-chef-57887
11/13/2023, 7:51 PM