do you have any examples of class-based tasks/workflows instead of functions?
or any examples of huge configs (like 100 options) with overload? usually we do it with hydra, but I wonder if there are any other options
another question: how is the data passed between tasks, if I have huge dataset to supply for model? (in luigi we usually pass filepaths even if the file is in s3, any best practices for flyte?)
f
freezing-airport-6809
05/02/2023, 2:51 PM
Hi @miniature-zebra-33903 firstly welcome to the community.
Large data is automatically passed using references to a blob or underlying store that fsspec supports.
This should explain how data is handled https://docs.flyte.org/en/latest/concepts/data_management.html
We will share a class based example - this is the issue that folks have asked - please explain your usecase as well https://github.com/flyteorg/flyte/issues/3492
👍 1
freezing-airport-6809
05/02/2023, 2:52 PM
Also you can use dataclasses to pass large data objects or any other objects