#6455 [Core feature] Allow using s5cmd for get/put operations on FlyteFile
Issue created by
ggydush
Motivation: Why do you think this is important?
s5cmd offers significant performance improvements over s3fs/aws s3 tools. In one instance, FlyteFile took roughly 20 minutes to download a file vs 2 minutes with s5cmd.
Goal: What should the final outcome look like, ideally?
s5cmd used as primary way to upload/download files with FlyteFile/FlyteDirectory
Describe alternatives you've considered
Abandoned usage of FlyteFile for heavy data tasks and opt to just run s5cmd via subprocess.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
• Yes
Have you read the Code of Conduct?
• Yes
flyteorg/flyte