#3005 [Core feature] Skyplane flytekit plugin
Issue created by
pingsutw
Motivation: Why do you think this is important?
SkyPlane make transferring data much faster and cheaper. Currently, we use awscli to upload/download data from s3, and awscli doesn't have a good performance on I/O. However, We can probably reduce overhead on I/O if we replace awscli with Skyplane.
We could add a Skyplane flytekit plugin first, and extend
DataPersistence to implement a new persistence plugin.
Finally, we should test it to see how much time we can save on I/O.
Goal: What should the final outcome look like, ideally?
Use Skyplane to upload / download the flyte literal by default if people install the Skyplane plugin.
Describe alternatives you've considered
Use awscli, which is what we already have now.
Propose: Link/Inline OR Additional context
Skyplane: 110x faster data transfers on any cloud
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte