What is the "flyte way" to handle workflows with a lot of data? We have several workflows that handles 10s of GBs, and they're failing due to the output size -

is too large [28775519] bytes, max allowed [10485760] bytes

For now, we're passing a Flytefile instead of the actual data to overcome this issue, and I also understand another approach could be increasing the

max-output-size-bytes

parameter, but this is only temporary, as data in the future could succeed this threshold. So - What is the proper way to handle large I/O?

What is this data that you return that is unlined and 10gb

lists of jsons (dicts)

Aah yes JSON is passed inline

hundred of thousands jsons.. is there maybe a different data type that you support that will offload?

We have never seen a json list of 10gb

I'll try using a dataframe, but I wish it could be supported with some data type that supports typing hints.. could really help us (like list of dicts where I can define the schema)

If you are going to go with a dataframe, you can look at StructuredDatasets, they support strongly typed schemas that are compile-type validated... You can even combine it with Pandera to define rules around validation and have flyte automatically kick these off.

What about Pydantic data models? Is there a place I can see which types are transfered in-line and which are offloaded?

https://flyte-org.slack.com/archives/C06H1SFA19R/p1727112164629489?thread_ts=1727112163.247309&cid=C06H1SFA19R This is probably the closest to what you are looking for: https://docs.flyte.org/en/latest/concepts/data_management.html