I had a few workflows with map tasks panic with this error:

RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: panic when reconciling workflow, Stack: [goroutine 839 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow.func2.1()
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:137 +0x6e
panic({0x2517d20?, 0xc01b420a20?})
	/usr/local/go/src/runtime/panic.go:792 +0x132
github.com/flyteorg/flyte/flytestdlib/bitarray.(*CompactArray).validateValue(0xc001654090?, 0x30e7460?)
	/go/src/github.com/flyteorg/flytestdlib/bitarray/compact_array.go:34 +0xad
github.com/flyteorg/flyte/flytestdlib/bitarray.(*CompactArray).SetItem(0xc003b9f4b0, 0x64, 0x417db)
	/go/src/github.com/flyteorg/flytestdlib/bitarray/compact_array.go:41 +0x4e
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/array.(*arrayNodeHandler).Handle(...
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/array/handler.go:422 +0x4909
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).execute(...
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:855 +0x12f
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleQueuedOrRunningNode(...
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:1159 +0xa7d
...
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).runWorker(...
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:116 +0x9a
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Run.func1()
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:151 +0x4f
created by github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Run in goroutine 531
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:148 +0x265
]

It boiled down to a parameter called `MaxDeltaTimestamp` with description

json:"max-delta-timestamp" pflag:",Maximum delta timestamp between ArrayNode start and an individual subNode start."

having a too low value by default (3d), causing the overflow. The bit array is allocated here. The reason was that the tasks used spot instances and even after 3 days, a task might have a new attempt. I feel for users this is likely a rather hard bug to understand. Do you think there are performance issues if we increase this to a higher number of days by default to reduce the likelyhood of people running into this? (Handling it more gracefully in general would of course be best.)