#1512 Stream Directories and Files using Flyte Pull request opened by kumare3 TL;DR This PR implements streaming FlyteFile and allows walking FlyteDirectory. the streaming FlyteFile returns a Filelike object that is supported by fsspec. Note this is still experimental and we would love feedback on the API. Example: Accept a flytefile and stream to another FlyteFile, using simple caching

@task
def copy_file(ff: FlyteFile) -> FlyteFile:
    new_file = FlyteFile.new_remote_file(ff.remote_path)
    with ff.open("r", cache_type="simplecache", cache_options={}) as r:
        with new_file.open("w") as w:
            w.write(r.read())
    return new_file

Example: Accept a flyteDirectory and walk it, while copying the directory file by file to another Flyte directory

@task
def process_folder(fd: FlyteDirectory) -> FlyteDirectory:
    out_fd = FlyteDirectory.new_remote()
    for base, x in fd.crawl():
        src = os.path.join(base, x)
        out_file = out_fd.new_file(x)
        with FlyteFile(src).open("rb") as f:
            with out_file.open("wb") as o:
                o.write(f.read())
    return out_fd

Alternatively, Files can always be handled by fsspec streaming

@task
            def copy_file(ff: FlyteFile) -> FlyteFile:
                new_file = FlyteFile.new_remote_file(ff.name)
                fs = flytekit.get_filesystem()
                with fs.open(f"readahead::{ff.remote_path}", "rb", readahead={}) as r:
                    with new_file.open("wb") as w:
                        w.write(r.read())
                return new_file

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description Other minor changes: • Get rid of sqlite data download dependency by adding it to this repo • Make the base to_html function not abstract. Tracking Issue flyteorg/flyte#3334 flyteorg/flytekit GitHub Actions: lint GitHub Actions: build-plugins (3.10, flytekit-vaex) GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.10, flytekit-spark) GitHub Actions: build-plugins (3.10, flytekit-snowflake) GitHub Actions: build-plugins (3.10, flytekit-ray) GitHub Actions: build-plugins (3.10, flytekit-polars) GitHub Actions: build-plugins (3.10, flytekit-papermill) GitHub Actions: build-plugins (3.10, flytekit-pandera) GitHub Actions: build-plugins (3.10, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.10, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.10, flytekit-kf-mpi) GitHub Actions: build-plugins (3.10, flytekit-k8s-pod) GitHub Actions: build-plugins (3.10, flytekit-hive) GitHub Actions: build-plugins (3.10, flytekit-duckdb) GitHub Actions: build-plugins (3.10, flytekit-dolt) GitHub Actions: build-plugins (3.10, flytekit-deck-standard) GitHub Actions: build-plugins (3.10, flytekit-dbt) GitHub Actions: build-plugins (3.10, flytekit-data-fsspec) GitHub Actions: build-plugins (3.10, flytekit-dask) GitHub Actions: build-plugins (3.10, flytekit-bigquery) GitHub Actions: build-plugins (3.10, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.10, flytekit-aws-batch) GitHub Actions: build-plugins (3.10, flytekit-aws-athena) GitHub Actions: build-plugins (3.9, flytekit-whylogs) GitHub Actions: build-plugins (3.9, flytekit-vaex) GitHub Actions: build-plugins (3.9, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.9, flytekit-spark) GitHub Actions: build-plugins (3.9, flytekit-snowflake) ✅ 1 other check has passed 1/30 successful checks