#253 [Breaking] Obstore parallel reader Pull request opened by wild-endeavor Just copying from @jeevb. Thanks! The obstore store object by default uses a single connection - this PR introduces a parallel reader (writer to come) to the storage layer, to read files in parallel. Files under the default chunk size of 16MBs will continue to just use one connection. Also using this opportunity to update the API around how the offloaded types (files/dir) behave. The new behavior better mirrors how

aws s3 cp

works. ## Breaking ### File objects If you have a task that takes in some

f: flyte.io.File

with a

path

set to

<s3://bucket/file.txt>

, the behavior for

download

now is •

f.download()

without an argument for the local destination - will download to a randomly created local folder with a filename of

file.txt

•

f.download("some/filename")

with a file name specified - will download to that specified file name. •

f.download("new/path/")

or

f.download("existing/folder")

with an existing local directory or with a non-existent string that ends with

/

- will download to

file.txt

under that folder name, creating it if necessary. #### file.open() This was introduced in another PR, but will mention it again here as some examples were updated. If you have a file object, and you do

async with f.open() as fh

, you'll now need to do

await fh.read()

instead of just

fh.read

. Also, the output of

fh.read

(when using s3/gcs/abfs) will be an obstore Bytes object, not a python bytes object, which means if you are calling

.decode

on it directly it will fail. If you're trying to read it as a string, use

str(b, "utf-8")

rather than calling

to_bytes

on it to save the memory copy. ### Dir objects If you have a task that takes in some

d: flyte.io.Dir

with a

path

set to

<s3://bucket/my_remote_folder>

, the behavior for download is now •

d.download()

The contents of

my_remote_folder

will be downloaded into a new local directory. •

d.download("new_parent/new_folder")

The contents of

my_remote_folder

will be downloaded as the contents of

new_parent/new_folder

•

d.download("existing_dir")

The contents of

my_remote_folder

will be downloaded as the contents of

existing_dir

In no case will the name of the remote folder

my_remote_folder

show up. Previously this was not the case and the remote folder name would show up locally. The new behavior matches what

aws s3 cp --recursive

does and also mostly matches *nix

cp -R

. Also, • The

remote_path

input to

Dir.from_local

has been changed to

remote_destination

to match the

File

object. flyteorg/flyte-sdk