Hi I have a question regarding data flow in flyte - I read the docs on this and I understand that flyte only passes around reference to files, but if I have a large dataset in s3, for example, does that mean each task that has to process/use that data will download the entire dataset onto each task container’s memory?

yes, you could either download entire dataset or read file steam

OK thanks! If the dataset is really large though, wouldn’t this take a long time to download and process the dataset for each task?

if you use tensorflow or pytorch, they have some features that allow you train model with batch data. it means it can download and process data in the same time.

That’s interesting but a more basic question first - if I have image data in s3 and I supply the s3 url as task input, I still need a logic to download the dataset in the task code right (e.g. via

requests

lib, etc)?. And then if I process it and send it to another task, then flyte will take care of creating the reference for the s3 data and I can just use it in the second task as if it’s already in memory?

flyte uses lazy download. it downloads the file only when you call

file.download()

,

file.open()

or

__fspath__

Oh ok didn’t know that - thanks!

OK will check it out thank you!

No, only the task specified storage

OK so if I have

preprocess_task

->

train_task

, and download in

preprocess_task

to do some data processing and pass it over to

train_task

, then will Flyte basically upload the processed data from the first task to s3 and then download that data in

train_task

?

Yes, correct

Ok I see - so currently no funtionality for sharing storage between tasks then