Pradithya Aria Pura
05/16/2023, 6:02 AMFLYTE_GCS_GSUTIL_PARALLELISM
to true
?FlyteDirectory
transfer when there are a lot of files ~10k and just realized the parallelism is disabled by default. What will be the best approach to enable it globally? cc: @Lee Ning Jie LeonKetan (kumare3)
jeev
Pradithya Aria Pura
05/16/2023, 3:15 PMjeev
Pradithya Aria Pura
05/16/2023, 3:20 PMjeev
Pradithya Aria Pura
05/16/2023, 3:24 PMjeev
Pradithya Aria Pura
05/16/2023, 3:25 PMKetan (kumare3)
Pradithya Aria Pura
05/16/2023, 3:26 PMKetan (kumare3)
Pradithya Aria Pura
05/17/2023, 3:22 AMjeev
Ketan (kumare3)
Pradithya Aria Pura
05/17/2023, 3:52 AMIn the new version @Pradithya Aria Pura I would recommend using the streaming apiNoted will keep this in mind. Thanks @jeev this is really useful!