In the older version of flytekit pre 1 5 we can configure gs

Question

In the older version of flytekit pre 1 5 we can configure gsutil parallelism using <https github com flyteorg flytekit blob d23df46a68a30cc09210589831941bdc0e4d08de flytekit configuration internal py LL57C1 L58C1|this configuration> Is it correct to assume that I can set the value by setting environment variable `FLYTE GCS GSUTIL PARALLELISM` to `true`

Pradithya Aria Pura · Answer

Asking since we see a slow `FlyteDirectory` transfer when there are a lot of files ~10k and just realized the parallelism is disabled by default What will be the best approach to enable it globally cc < Lee Ning Jie Leon>

Ketan (kumare3) · Answer

Ohh now we use fsspec I would have expected it to be faster Cc < Yee> < jeev>

jeev · Answer

< Pradithya Aria Pura> what flytekit version

Pradithya Aria Pura · Answer

I am using 1 2 11 still stuck with lt 1 3 due to protobuf version

jeev · Answer

If y all are still using gsutil under the hood for data persistence <https github com flyteorg flytekit blob d23df46a68a30cc09210589831941bdc0e4d08de flytekit configuration internal py L57|https github com flyteorg flytekit blob d23df46a68a30cc09210589831941bdc0e4d08de flytekit configuration internal py L57>

jeev · Answer

FLYTE GCP GSUTIL PARALLELISM

Pradithya Aria Pura · Answer

Haven t tested it yet Still figuring out the approach to enable it globally so that all workflow will get the benefit Any suggestion

jeev · Answer

what about default env var in propeller plugin config

Pradithya Aria Pura · Answer

Do you mean <https docs flyte org en latest deployment configuration generated flytepropeller config html default env vars map string string|this>

jeev · Answer

yes

Pradithya Aria Pura · Answer

Got it thanks Will try and update in this thread

Ketan (kumare3) · Answer

So after flytekit 1 5 you should not need it

Pradithya Aria Pura · Answer

yeah hopefully we can reach to that point asap crossed fingers

Ketan (kumare3) · Answer

Also let us know about 1 5 and how it s working etc

Pradithya Aria Pura · Answer

It works And it scale with the number of CPU too Previously it tooks ~17minutes to copy 13k images now it s around ~5minutes with 4 CPU cores

jeev · Answer

you can further tune it too if you can mount a boto cfg into the container <https medium com duhroach gcs read performance of large files bd53cfca4410>

Ketan (kumare3) · Answer

In the new version < Pradithya Aria Pura> I would recommend using the streaming api

Ketan (kumare3) · Answer

The same method can be accelerate as it will use very little disk and memory

Pradithya Aria Pura · Answer

gt In the new version < Pradithya Aria Pura> I would recommend using the streaming api Noted will keep this in mind Thanks < jeev> this is really useful