most-gold-65483
05/16/2023, 6:02 AMFLYTE_GCS_GSUTIL_PARALLELISM
to true
?most-gold-65483
05/16/2023, 6:15 AMFlyteDirectory
transfer when there are a lot of files ~10k and just realized the parallelism is disabled by default. What will be the best approach to enable it globally? cc: @broad-train-34581freezing-airport-6809
freezing-boots-56761
most-gold-65483
05/16/2023, 3:15 PMfreezing-boots-56761
freezing-boots-56761
most-gold-65483
05/16/2023, 3:20 PMfreezing-boots-56761
most-gold-65483
05/16/2023, 3:24 PMfreezing-boots-56761
most-gold-65483
05/16/2023, 3:25 PMfreezing-airport-6809
most-gold-65483
05/16/2023, 3:26 PMfreezing-airport-6809
most-gold-65483
05/17/2023, 3:22 AMfreezing-boots-56761
freezing-airport-6809
freezing-airport-6809
most-gold-65483
05/17/2023, 3:52 AMIn the new version @most-gold-65483 I would recommend using the streaming apiNoted will keep this in mind. Thanks @freezing-boots-56761 this is really useful!