Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Also, <@U04664Z7H37> do you folks use - <https://github.com/libffcv/ffcv>?

created this - <https://github.com/flyteorg/flyte/issues/3615>

Mh no, not using it but looks interesting. <https://arxiv.org/abs/2209.13705|This> paper compares ffcv to other libraries including squirrel which my previous company built. The authors didn’t use many features of squirrel though, otherwise would be faster.

When I look at the code snippets on ffcv’s website+github, I’d say that this all should live in user code though, just needs an image with dependencies installed. Which job should Flyte take in your opinion?

&gt; For example, we could not run FFCV with a dataset hosted in an S3 bucket to perform our remote experiments.
(From the comparison paper)

This is a downside for a data loading library tbh. Squirrel from my previous company uses fsspec and was designed for remote loading.

Interesting that it would only work with local files