#6725 Fixed bug in loop index to properly mark items as processing Pull request opened by Sally-Yang-Jing-Ou ## Why are the changes needed? As part of the effort of investigating "parent workflows getting stuck despite subworkflows finished" (reported bug ) Bug i=1 Analysis Inside the

enqueueBatches

function of auto_refresh.go, the loop that adds items into the processing map incorrectly starts at

i = 1

rather than

i = 0

. Since our cache configuration batches one item per batch, this logic results in never inserting items into the processing map. Why this is a problem: The processing set is used to prevent enqueueing items that are already being processed. Since items are not being added to processing, every item is repeatedly re-added to the workqueue. Flyte uses Kubernetes'

workqueue.RateLimitingInterface

. When an item is re-enqueued via AddRateLimited, the queue computes a delay and stores it inside a delaying queue (

waitFor{item, readyAt}

). Items are only released for processing once

readyAt

is reached. Because items are repeatedly re-enqueued unnecessarily: • The delay grows over time (linear per queue rate limiter settings). • The internal delaying queue accumulates a backlog of delayed items • These delays can grow extremely large, causing items to take a long time to get on the immediate workqueue. ## What changes were proposed in this pull request? Fix the loop bug from

i=1

to

i=0

. ## How was this patch tested? Unit test is added. At LinkedIn, we were able to run stress testing of this case in two different cluster, one with the bug and one without the bug. Reproduction Experiment To reproduce the issue consistently: • Cache configuration: size=20000, tps=10, burst=10, workers=5 • Updated sync period frequency to 5 seconds to maximize enqueue pressure • Launched a workflow that spawns 100 subworkflows every 3 minutes for 15 hours (~30,000 subworkflows of reference launch plan total) We ran this experiment under two conditions: • Baseline: Original code (i = 1) • Test variant: Fixed logic (i = 0) Result: • Baseline exhibited indefinitely running behavior even though its subworkflows succeeded. • The test variant showed sustained progress, confirming that the loop bug was a contributing factor. ### Labels • fixed: For any bug fixed. ### Screenshots (hmm... couldn't paste screenshots here in the PR, told by some popup message since this is my first PR against Flyte upstream). ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. https://bito.ai#summarystart| ### Summary by Bito • Addresses a critical bug in the

enqueueBatches

function of the

in_memory_auto_refresh.go

file, where the loop index was incorrectly set to start at 1 instead of 0, preventing items from being properly marked as processing. • Introduces unit tests to verify the correct behavior for both single and multi-item batches, enhancing the reliability of the caching mechanism. • Overall summary: addresses a critical bug in the

enqueueBatches

function and introduces unit tests. flyteorg/flyte