Also, @Fabio Grätz do you folks use - https://github.com/libffcv/ffcv?

Mh no, not using it but looks interesting. This paper compares ffcv to other libraries including squirrel which my previous company built. The authors didn’t use many features of squirrel though, otherwise would be faster. When I look at the code snippets on ffcv’s website+github, I’d say that this all should live in user code though, just needs an image with dependencies installed. Which job should Flyte take in your opinion?

Interesting that it would only work with local files