Hi,
What is the correct Flyte way to return a list of files? (they are currently in a zip, and i want to split them into batches after). When i return List[FlyeFile] i get permission denied for the files (saving in /tmp)
When returning them from a task
f
freezing-airport-6809
11/06/2024, 3:46 PM
You can return a zip as a single file
freezing-airport-6809
11/06/2024, 3:46 PM
You can still split it later just return the zip for now
freezing-airport-6809
11/06/2024, 3:46 PM
Also list of Flytefile should work
l
lively-article-96254
11/06/2024, 3:50 PM
they are already in a zip, i want to unzip them to chunk into batches for a map_task.
A list of FlyeDIrectorys actually worked, not sure why a list of FlyteFiles didn't.
f
freezing-airport-6809
11/06/2024, 3:51 PM
Hmm weird - you can also use an iterator to stream write
l
lively-article-96254
11/06/2024, 3:52 PM
Interesting, and that will make each pod only download each needed batches?