Also, <@U04664Z7H37> do you folks use - <https://g...
# torch-elastic
k
Also, @Fabio Grätz do you folks use - https://github.com/libffcv/ffcv?
f
Mh no, not using it but looks interesting. This paper compares ffcv to other libraries including squirrel which my previous company built. The authors didn’t use many features of squirrel though, otherwise would be faster. When I look at the code snippets on ffcv’s website+github, I’d say that this all should live in user code though, just needs an image with dependencies installed. Which job should Flyte take in your opinion?
For example, we could not run FFCV with a dataset hosted in an S3 bucket to perform our remote experiments.
(From the comparison paper) This is a downside for a data loading library tbh. Squirrel from my previous company uses fsspec and was designed for remote loading.
k
Interesting that it would only work with local files
103 Views