Returning a FlyteDirectory from a ContainerTask is supported, but until recently, Flyte only supported downloading directories (inputs) as multipart blobs, not uploading them (outputs) from container tasks. As of PR #5715, Flyte supports multipart blob download for FlyteDirectory, but upload support is still incomplete. This means your ContainerTask can consume a FlyteDirectory as input, but returning a FlyteDirectory as output from a raw container is not fully supported yet—your output directory may not be uploaded as expected. For now, a workaround is to use a Python @task with an ImageSpec that wraps your container logic, or to use a ListFlyteFile] if possible. See the discussion and workaround in [issue #3632 and the current state in issue #5924. Would you like more detail or example code for the workaround? Sources: - PR #5715: multipart blob download for FlyteDirectory - Issue #5924: multipart blob upload not supported - Issue #3632: FlyteDirectory not copied to ContainerTask inputs 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.

The most reliable workaround is to use a Python @task with an ImageSpec that uses your desired container image as the base, and then implement the directory creation logic in Python. This way, Flytekit handles the FlyteDirectory serialization/deserialization, ensuring correct upload and download. For example:

from flytekit import task, workflow, ImageSpec
from flytekit.types.directory import FlyteDirectory

image_spec = ImageSpec(
    base_image="alpine:3.18",
    packages=["flytekit"],  # Ensure flytekit is installed in the image
)

@task(container_image=image_spec)
def produce_data() -> FlyteDirectory:
    import os
    out_dir = "/tmp/results"
    os.makedirs(out_dir, exist_ok=True)
    with open(os.path.join(out_dir, "hello.txt"), "w") as f:
        f.write("hello")
    return FlyteDirectory(path=out_dir)

This approach avoids the limitations of raw ContainerTask output handling for FlyteDirectory, as Flytekit manages the data movement. See issue #3632 for more discussion. Would you like a full workflow example or more details on customizing the image? Sources: - Issue #3632: workaround using Python task and ImageSpec 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

In that case, there is currently no fully supported way to return a FlyteDirectory from a raw ContainerTask, due to the lack of multipart blob upload support for directories in the sidecar/uploader (see issue #5924). The only available workarounds are to use a List[FlyteFile] as output (if you know the files in advance), or to wait for the FlyteDirectory upload feature to be implemented for ContainerTask outputs. If you must use a raw container and need directory output, you may need to restructure your workflow to output individual files or use another mechanism until this feature is available. See the discussion and limitations in issue #3632 and issue #5924. Sources: - Issue #5924: multipart blob upload not supported - Issue #3632: FlyteDirectory not copied to ContainerTask inputs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

It seems likely that Flyte users working with many small files (like images and numpy arrays) often group them into a single archive (e.g., .zip, .tar, or HDF5) to reduce the number of files handled by FlyteDirectory, since each file incurs overhead during upload/download and task setup/teardown. Grouping files can improve performance, as Flyte's data transfer is more efficient with fewer, larger files rather than thousands of small ones. This is also suggested in the context of Flyte's batch upload/download for FlyteDirectory and in discussions about performance bottlenecks with many files (see here, FlyteDirectory docs). Common approaches include using compressed archives (.zip, .tar.gz), HDF5 files for structured/unstructured data, or even big pickle files for Python-native objects. The choice depends on your downstream processing needs and compatibility. Reducing file count by grouping data will likely speed up both data transfer and task startup/teardown in Flyte. Would you like more detail or example code for any of these approaches? Sources: - see here - FlyteDirectory docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.