#1512 Stream Directories and Files using Flyte Pull request opened by kumare3 TL;DR This PR implements streaming FlyteFile and allows walking FlyteDirectory. the streaming FlyteFile returns a Filelike object that is supported by fsspec. Note this is still experimental and we would love feedback on the API. Example: Accept a flytefile and stream to another FlyteFile, using simple caching

@task
def copy_file(ff: FlyteFile) -> FlyteFile:
    new_file = FlyteFile.new_remote_file(ff.remote_path)
    with ff.open("r", cache_type="simplecache", cache_options={}) as r:
        with new_file.open("w") as w:
            w.write(r.read())
    return new_file

Example: Accept a flyteDirectory and walk it, while copying the directory file by file to another Flyte directory

@task
def process_folder(fd: FlyteDirectory) -> FlyteDirectory:
    out_fd = FlyteDirectory.new_remote()
    for base, x in fd.crawl():
        src = os.path.join(base, x)
        out_file = out_fd.new_file(x)
        with FlyteFile(src).open("rb") as f:
            with out_file.open("wb") as o:
                o.write(f.read())
    return out_fd

Alternatively, Files can always be handled by fsspec streaming

@task
            def copy_file(ff: FlyteFile) -> FlyteFile:
                new_file = FlyteFile.new_remote_file(ff.name)
                fs = flytekit.get_filesystem()
                with fs.open(f"readahead::{ff.remote_path}", "rb", readahead={}) as r:
                    with new_file.open("wb") as w:
                        w.write(r.read())
                return new_file

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description Other minor changes: • Get rid of sqlite data download dependency by adding it to this repo • Make the base to_html function not abstract. Tracking Issue flyteorg/flyte#3334 flyteorg/flytekit Codecov: 41.48% of diff hit (target 69.70%) Codecov: 69.31% (-0.39%) compared to 93995e2 ✅ 28 other checks have passed 28/30 successful checks