I'm not sure how much of an antipattern this is, but I've been thinking about creating an EFS based PVC for each flyte namespace and using it to pass large objects rather than having to do a S3 upload/download between tasks. It breaks the idea of tasks being fully independent outside of the DAG connection, but I'm building a LMDB for ML inference in one task and then passing it to a GPU task for inference and IO on 500G files can be quite slow

This is fine

It would be faster if only because it's write->read rather than write->upload->download->read

ohh you dont have to upload /download thing. you can simply stream right?

Between tasks? Isn't it a forced S3 IO step?

@freezing-tailor-85994 - https://www.union.ai/docs/flyte/user-guide/data-input-output/flyte-file-and-flyte-directory/#streaming-support