#6188 [BUG] Memory usage regression in tasks using FlyteFiles in flytekit v.14.X
Issue created by
Sovietaced
Describe the bug
After upgrading to Flyte v1.14.X we noticed that tasks that previously consumed ~1GB of memory started to consume much more memory (and lead to OOMs at 16GB). We've found this to happen on tasks with the same inputs/datasets.
Memory Usage on v1.13.X
Image
Memory Usage on v.14.X
Image
Expected behavior
Memory usage is similar to v1.13.X
Additional context to reproduce
Noticed with tasks using large amounts of FlyteFiles (~21k) sized between 2-3MB after the task returns.
Screenshots
No response
Are you sure this issue hasn't been raised already?
• Yes
Have you read the Code of Conduct?
• Yes
flyteorg/flyte