Hi, is there a way to allow empty FlyteDirectories? From the concept of a directory, I would expect empty directories to be a valid thing (even if the storage backend is S3). With the current implementation, it fails at calling .download()

this doesn’t sound unreasonable at least. sounds like just a corner case that was missed. what’s the use-case though?

It is indeed a corner case, we have tasks in our workflows that return a FlyteDirectory, which can be empty - an example would be a task that creates visualizations of bounding boxes from a previous object detection stage - if there are no bounding boxes, this FlyteDirectory would be empty. Optional is the way we handle things currently, but it would make things a bit cleaner, if FlyteDirectory could handle this internally (e.g. returning a path to an empty folder when calling .download(). If this is just a missed corner case, great - then I will add it to the list of potential UX improvements (https://github.com/flyteorg/flyte/issues/4542). After reading this discussion, I had the impression, this design was on purpose: https://github.com/flyteorg/flyte/discussions/3502#discussioncomment-5617773

Sadly a directory in a blob store is a prefix path. And empty keys are not possible- we could write a dummy file, but that would break contract

On the remote storage it is absolutely fine, but the FlyteDirectory itself could store a state (is_empty=True) and when calling .download() on the downstream task simply return an empty directory on the local filesystem

Ya,