. This slows down serialization. it takes more than 15 mins to upload the pickles to s3 if the size of list is 1000.
People don't care about how we serialize
List[Any]
. We can just convert entire list into a single pickle file, which reduces the time required for serialization.
Goal: What should the final outcome look like, ideally?
it will make serialization faster
Describe alternatives you've considered
• Raise an error when using large list
• Add a detailed warning
Propose: Link/Inline OR Additional contextSlack ThreadAre you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte