<#6802 Prevent panic when retry attempt value in m...
# flyte-github
a
#6802 Prevent panic when retry attempt value in map task gets larger than bitarray was allocated for Pull request opened by fg91 ## Why are the changes needed? With a default configuration of
max-node-retries-system-failures=3
and
default-max-attempts=1
, the following workflow leads to a panic in the node array handler: @task( requests=Resources(cpu="50m", mem="250Mi"), limits=Resources(cpu="100m", mem="250Mi"), retries=8, ) def test(inp: int) -> None: if inp == 2: # Go OOM on purpose or any other "retriable user error" a = " " * (300 * 1024 * 1024) @workflow def wf1() -> None: map_task(test)(inp=[i for i in range(3)]) The panic occurs here when validating whether the current value of attempts can be tracked in a bitarray that was allocated for this purpose. The following goes wrong: Currently the
retries=8
set on the task decorator is not taken into account when determining the max value of the number of attempts: Here in the array node handler, for the example workflow above,
nCtx.Node().GetRetryStrategy()
is
nil
.
maxAttemptsValue
is therefore
max-node-retries-system-failures=3
+
default-max-attempts=1
equals to
maxAttemptsValue=4
. When creating the bit array to track the retry attempts here, it is determined that 3 bits are needed to store values that cannot be higher than 4. However, the task is actually retried 8 times. Up to attempt 7 this still works because only 3 bits are needed. But attempt 8 then panics here. ## What changes were proposed in this pull request? When determining the max value that can be stored in the bit array, we need to account not only for
nCtx.Node().GetRetryStrategy()
but also for
nCtx.Node().GetArrayNode().GetSubNodeSpec().GetRetryStrategy()
which does contain the desired
retries=8
. During my debugging,
nCtx.Node().GetRetryStrategy()
was actually always
nil
but I cannot say for sure this is always the case. So I propose to allocate the bit array to have space for the max of both values to be on the safe side. ## How was this patch tested? The example above panics on latest master but works with the changes in this PR. ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. https://bito.ai#summarystart| ### Summary by Bito • Addresses a critical bug in the node array handler that could lead to a panic when the retry attempt value exceeds the allocated bit array size. • Ensures that both the node's retry strategy and the sub-node's retry strategy are considered when determining the maximum number of attempts. • Enhances the stability of workflows that involve retries, particularly in scenarios where the retry count is high. • Overall, this pull request addresses a critical bug in the node array handler, improving stability in retry workflows. flyteorg/flyte