Hey dear community ! We're trying to resolve a key issue that we currently have using flyte - passing large datasets between tasks. Our data comes in a complex json form, and we want to maintain a schema for the data. At first we tried to pass the data as python dicts - but we soon got errors due to the size of the data and the fact that dicts are passed inline ("output file is too large error") We tried using pydantic models and dataclass, but got the same error.. we're now trying with StructuredDataset, but it's less suitable for us due to the complex and nested schema. What is the best "flyte" way to pass large data sets between tasks in a way where we can maintain a schema?

How much data is this

Sure 🙂 !