#4542 [Core feature] FlyteFile UX Improvements Issue created by EngHabu ### Motivation: Why do you think this is important? FlyteFile & FlyteDirectory are the primary ways to access unstructured datasets usually stored remotely (s3, gs, az... etc.). The UX for interacting with these files is quite cumbersome IMHO. I wanted to start a collection of proposals and use-cases to improve the UX. If you have others to add, please comment below... ### Goal: What should the final outcome look like, ideally? 1. Be able to operate on FlyteFiles on the metadata level (e.g. inspect file metadata and group files based on creation data or a specific tag on the files) 2. Create

FlyteFile

from a remote url. This is supported but counter intuitive (

FlyteFile(path="s3://....", remote=False)

) 3.

FlyteDirectory.crawl()

should yield `FlyteFile`s and they should have metadata 4.

FlyteDirectory.crawl(details=True)

returns a list of tuples

(file_base, {file_path: dict})

a dict of one item is useless in my opinion. It should return

(file_base, (file_path, dict))

instead... but we should return FlyteFiles anyway and have materialized metadata on FlyteFiles.. maybe something like

(file_base, (file_path, FlyteFile))

5.

FlyteFile.download()

downloads files in random locations. I think these should be under the working directory by default (maybe in randomly generated folder names)... and I think it should allow you to download to a specific location... right now I've to

open(my target)

and write to it ### Describe alternatives you've considered Reimplement these at the task level. ### Propose: Link/Inline OR Additional context No response ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte