do you have any examples of class-based tasks/work...
# ask-the-community
a
do you have any examples of class-based tasks/workflows instead of functions? or any examples of huge configs (like 100 options) with overload? usually we do it with hydra, but I wonder if there are any other options another question: how is the data passed between tasks, if I have huge dataset to supply for model? (in luigi we usually pass filepaths even if the file is in s3, any best practices for flyte?)
k
Hi @Anatolii Bardukov firstly welcome to the community. Large data is automatically passed using references to a blob or underlying store that fsspec supports. This should explain how data is handled https://docs.flyte.org/en/latest/concepts/data_management.html We will share a class based example - this is the issue that folks have asked - please explain your usecase as well https://github.com/flyteorg/flyte/issues/3492
Also you can use dataclasses to pass large data objects or any other objects
155 Views