I have a question related to automatic downloads of FlyteFiles/FlyteFolders. I have a large amount of data (~1TB) spread across ~15,000 files stored in GCS. I have a Flyte workflow that, as a first step, downloads these files to be operated on locally. I noticed that the download speed is around 20 MB/s and uses about 1-2 CPU cores. I did some testing, and when I manually invoke

gcloud storage cp -r …

inside of a Flyte workflow, I am able to download the same exact files at approximately 1.7GB/s, and I see usage of ~20 CPU cores (the workflow requests 32 cpu cores). I observe the same behavior when uploading files to GCS as well. How can I still make use of FlyteFiles/FlyteFolders and the automatic download/upload functionality, but get faster download/upload speed?

which version of flytekit are you on?

I’m not sure… how do I find out?

just

pip show flytekit

I think 1.2.7

would you be able to bump to a newer version and try again?

Yeah I saw that in the docs, I don’t have the ability to upgrade though, that decision is made outside my team

cc @freezing-boots-56761 😄

lol

can you bump to 1.5 please?