https://flyte.org logo
#ask-the-community
Title
# ask-the-community
k

Klemens Kasseroller

12/19/2023, 10:30 AM
Hi, is there a way to allow empty FlyteDirectories? From the concept of a directory, I would expect empty directories to be a valid thing (even if the storage backend is S3). With the current implementation, it fails at calling .download()
y

Yee

12/19/2023, 11:39 AM
this doesn’t sound unreasonable at least. sounds like just a corner case that was missed. what’s the use-case though?
is this something that an optional FlyteDirectory would work for?
k

Klemens Kasseroller

12/19/2023, 12:47 PM
It is indeed a corner case, we have tasks in our workflows that return a FlyteDirectory, which can be empty - an example would be a task that creates visualizations of bounding boxes from a previous object detection stage - if there are no bounding boxes, this FlyteDirectory would be empty. Optional is the way we handle things currently, but it would make things a bit cleaner, if FlyteDirectory could handle this internally (e.g. returning a path to an empty folder when calling .download(). If this is just a missed corner case, great - then I will add it to the list of potential UX improvements (https://github.com/flyteorg/flyte/issues/4542). After reading this discussion, I had the impression, this design was on purpose: https://github.com/flyteorg/flyte/discussions/3502#discussioncomment-5617773
k

Ketan (kumare3)

12/19/2023, 3:29 PM
Sadly a directory in a blob store is a prefix path. And empty keys are not possible- we could write a dummy file, but that would break contract
k

Klemens Kasseroller

12/19/2023, 3:44 PM
On the remote storage it is absolutely fine, but the FlyteDirectory itself could store a state (is_empty=True) and when calling .download() on the downstream task simply return an empty directory on the local filesystem
k

Ketan (kumare3)

12/19/2023, 3:48 PM
Ya,
In our metadata - absolutely