https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Anatolii Bardukov

05/02/2023, 7:54 AM
do you have any examples of class-based tasks/workflows instead of functions? or any examples of huge configs (like 100 options) with overload? usually we do it with hydra, but I wonder if there are any other options another question: how is the data passed between tasks, if I have huge dataset to supply for model? (in luigi we usually pass filepaths even if the file is in s3, any best practices for flyte?)
k

Ketan (kumare3)

05/02/2023, 2:51 PM
Hi @Anatolii Bardukov firstly welcome to the community. Large data is automatically passed using references to a blob or underlying store that fsspec supports. This should explain how data is handled https://docs.flyte.org/en/latest/concepts/data_management.html We will share a class based example - this is the issue that folks have asked - please explain your usecase as well https://github.com/flyteorg/flyte/issues/3492
Also you can use dataclasses to pass large data objects or any other objects
68 Views