Not sure if it'd speed things up, but probably would help you avoid the error

Regardles, the

.download

should manage the concurrency better and not allow the

RequestTimeTooSkewed

- my guess is that underneath the hood lots of

async

coroutines are spawned and some of them timeout somehow.

And parallelize it

Request time skew is indeed weird. Cc @thankful-minister-83577 have you seen this.

@flat-waiter-82487 I would have loved to give you v2, as the system has so much better control over io.

You could walk the directory

[3:35 PM]

And parallelize it

But that's manual, the FlyteDirectory abstractions should do it by itself 😄

If I have to walk the dir, I don't need FlyteDirectory at all

It’s also using fsspec underneath

(it's not the walk, it's more likely something underneath that downloads - I've seen some async mumbo jumbo underneath and I decided not to spend much time unrolling it - there's this

loop_manager

thing and tbh I don't know when "await" is happening - are all files discovered during "walk" scheduled at once and then just awaited? Is there a control of the concurrency there?)

We use this to download 70k+ files just fine

I have come across that error before tho 😬