#3347 FlyteRemote - Delay fetching task execution data until node complete Pull request opened by wild-endeavor ## Tracking issue Internal issue: https://linear.app/unionai/issue/BB-6143/missing-outputs-when-syncing-sbworkflow-in-flyteremote-execution ## Why are the changes needed? FlyteRemote was failing with

failed to fetch object: rpc error: code = NotFound desc = rpc error: code = NotFound desc = request failed with status code 404. Body: {"code":5, "message":"object '<s3://union-bucket/metadata/propeller/flytesnacks-development-a78lg8z5tpxznp2nf79j/n0/data/0/outputs.pb>' not found

The reason this happens is because propeller is writing the outputs uri immediately after the dynamic task itself is done, not waiting until the dynamic subworkflow has been completed. Admin checks that the output uri is there before it tries to fetch but since it's getting set by propeller, it just returns a 404. Notably,

FlyteRemote

is falling into the 'normal-node' section of sync_node_execution because the node execution metadata for the dynamic node isn't set initially (i.e. is_parent_node is False and is_dynamic is False). Only later is it set - so I think the race is between the node getting these flags set and the task execution for the dynamic task itself getting marked as succeeded. ## What changes were proposed in this pull request? Delay fetching task execution data until node is complete. ## How was this patch tested? Made a dummy workflow that was able to repro the error maybe half of the time and added logic to fix. Confirmed with debugging logic now removed that the race condition was hit and ignored. ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link https://bito.ai#summarystart| ### Summary by Bito • This pull request addresses a race condition that caused 404 errors when attempting to access output data prematurely by delaying the fetching of task execution data until the associated node is complete. • The modifications ensure that the task execution data is only retrieved when it is guaranteed to be available, enhancing the reliability of the workflow execution process. • Overall, this change improves the FlyteRemote functionality by modifying the task execution data retrieval process to prevent errors. flyteorg/flytekit