Anatolii Bardukov

05/02/2023, 7:54 AM
do you have any examples of class-based tasks/workflows instead of functions? or any examples of huge configs (like 100 options) with overload? usually we do it with hydra, but I wonder if there are any other options another question: how is the data passed between tasks, if I have huge dataset to supply for model? (in luigi we usually pass filepaths even if the file is in s3, any best practices for flyte?)

Ketan (kumare3)

05/02/2023, 2:51 PM
Hi @Anatolii Bardukov firstly welcome to the community. Large data is automatically passed using references to a blob or underlying store that fsspec supports. This should explain how data is handled We will share a class based example - this is the issue that folks have asked - please explain your usecase as well
Also you can use dataclasses to pass large data objects or any other objects