#5349 [BUG] Exception thrown when waiting for execution to finish Issue created by ggydush ### Describe the bug The following exception is thrown sometimes when executing a workflow with a dynamic that spins up multiple tasks: elif e.code() == grpc.StatusCode.NOT_FOUND:

raise FlyteEntityNotExistException() from e

E flytekit.exceptions.user.FlyteEntityNotExistException: USEREntityNotExist error=None, cause=<_InactiveRpcError of RPC that terminated with: E status = StatusCode.NOT_FOUND E details = "failed to fetch object: rpc error: code = NotFound desc = request failed with status code 404. Body: {"code":5,"message":"object 's3://union-compute-us-east-1-delvebio/metadata/propeller/clinical-pipeline-development-f76e3bd80dbf548ae8dc/n1/data/0/n3/0/outputs.pb' not found","details":[]}" E debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-05-10T123504.72984-07:00", grpc_status:5, grpc_message:"failed to fetch object: rpc error: code = NotFound desc = request failed with status code 404. Body: {\"code\":5,\"message\":\"object \'s3://union-compute-us-east-1-delvebio/metadata/propeller/clinical-pipeline-development-f76e3bd80dbf548ae8dc/n1/data/0/n3/0/outputs.pb\' not found\",\"details\":[]}"}" E > ### Expected behavior No exception when waiting for execution ### Additional context to reproduce execution = remote.execute( name, inputs=inputs, version=version or None, **kwargs, ) remote.wait(execution) ### Screenshots No response ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte