seunggs
04/21/2023, 3:59 PMKevin Su
04/21/2023, 4:10 PMseunggs
04/21/2023, 4:12 PMKevin Su
04/21/2023, 5:23 PMseunggs
04/21/2023, 5:38 PMrequests
lib, etc)?. And then if I process it and send it to another task, then flyte will take care of creating the reference for the s3 data and I can just use it in the second task as if it’s already in memory?Kevin Su
04/21/2023, 6:06 PMfile.download()
, file.open()
or __fspath__
file.open()
return file handler, and you can read streaming dataseunggs
04/21/2023, 6:18 PMKevin Su
04/21/2023, 10:03 PMseunggs
04/21/2023, 10:05 PM@task
, do all flyte tasks in the same workflow share that storage?Kevin Su
04/23/2023, 1:01 AMseunggs
04/23/2023, 1:04 AMpreprocess_task
-> train_task
, and download in preprocess_task
to do some data processing and pass it over to train_task
, then will Flyte basically upload the processed data from the first task to s3 and then download that data in train_task
?Kevin Su
04/23/2023, 1:04 AMseunggs
04/23/2023, 1:05 AM