cuddly-jelly-27016
01/23/2025, 7:48 PMFlyteDirectory
doesn't support os.listdir
with a remote S3 path (fortunately, a local path works!). The error is shown as follows:
Screenshot 2025-01-08 at 9 10 58 PM
Though FlyteDirectory
provides listdir
method itself, I just think os.listdir
could provide a better UX.
Expected behavior
A FlyteDirectory
with a remote S3 path can be walked through by os.listdir
, as commented here
Additional context to reproduce
Create a demo
dir on minio and add a text file first. Then, run the following script:
import os
from flytekit import task
from flytekit.types.directory import FlyteDirectory
S3_DIR = "s3://my-s3-bucket/demo
@task
def create_fd() -> FlyteDirectory:
fd = FlyteDirectory(path=S3_DIR)
entities = os.listdir(fd)
for e in entities:
# Suppose there's no subdir
with open(os.path.join(S3_DIR, e), "r") as f:
print(f.read())
return fd
if name == "__main__":
fd = create_fd()
print(fd)
Screenshots
No response
Are you sure this issue hasn't been raised already?
• Yes
Have you read the Code of Conduct?
• Yes
flyteorg/flytecuddly-jelly-27016
01/23/2025, 7:48 PM