#3036 [Core feature] [Flytekit] Implement Multipart upload and download for flyteplugins-vaex Issue created by ryankarlos ### Motivation: Why do you think this is important? vaex plugin currently implemented flyteorg/flytekit#1230 only supports writing chunks to single file using

df.export(...)

as explained in the docs. We would ideally also like to support cases where if the dataframe is large, we can export the chunks in parallel to multiple parts using

df.export_many(...)

as demostrated in the vaex api docs https://vaex.readthedocs.io/en/docs/api.html#vaex.dataframe.DataFrameLocal.export_many ### Goal: What should the final outcome look like, ideally? if df chunksize is greater than some threshold (1M ?) than should serialise to blob using

df.export_many(...)

, otherwise default to current implementation of

df.export

https://github.com/flyteorg/flytekit/blob/8ae879eb379acf2e0b4923f1b0c855d01a1f14e5/plugins/flytekit-vaex/flytekitplugins/vaex/sd_transformers.py#L29-L38 When decoding use

df.open_many(...)

if dir has multiple files/parts (use glob pattern) or

df.open(...)

(as currently implemented) if just single part. https://github.com/flyteorg/flytekit/blob/8ae879eb379acf2e0b4923f1b0c855d01a1f14e5/plugins/flytekit-vaex/flytekitplugins/vaex/sd_transformers.py#L51-L57 ### Describe alternatives you've considered N/A ### Propose: Link/Inline OR Additional context See discussion in this PR flyteorg/flytekit#1230 (comment) ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte