#460 Download file in parallel Pull request opened by pingsutw This pull request refactors the

examples/stress/benchmark/large_io_comparison.py

benchmarking script to make it more flexible and maintainable. The main improvements include parameterizing resource allocations, making benchmark tests optional, and enhancing the HTML report generation to be more robust to missing data. Additionally, a minor change is made in the storage module to adjust a condition for bypassing obstore logic. Benchmarking script improvements: • Resource allocation for CPU and memory is now parameterized using the

CPU

and

MEMORY

constants, replacing hardcoded values throughout the benchmark tasks and environments. [1] [2] [3] [4] [5] • The

benchmark_all

function now accepts

test_file

and

test_dir

boolean parameters, allowing selective execution of file and directory benchmarks. This is reflected in both the function signature and the script entry point. [1] [2] • The

generate_benchmark_report

function and its usage have been updated to handle cases where some benchmark results may be missing (

None

), making the HTML report generation more robust and flexible. [1] [2] [3] [4] [5] • The HTML report footer now dynamically displays the number of iterations and the actual CPU/memory used, improving accuracy and clarity. Other changes: • In

src/flyte/storage/_storage.py

, the condition for bypassing obstore logic no longer requires the

recursive

flag, potentially broadening the scenarios where the bypass is used. Before: Screenshot 2026-03-05 at 1 59 20 PM After: Screenshot 2026-03-05 at 1 50 29 PM flyteorg/flyte-sdk