#3665 [BUG] Flytekit 1.5.0 uses far more RAM than flytekit 1.4.2 Issue created by ossareh Describe the bug I've documented this in slack: https://flyte-org.slack.com/archives/CREL4QVAQ/p1683729924011689 Duplicated here: For the same set of inputs with the only difference between: • flytekit version • schema change (container tag + k8s resource requests) Flytekit 1.5.0 uses FAR more RAM. Based on the changelog my best guess is that s3fs is using memory where the older mechanism in 1.4.2 was using disk. This seems like a bad default when considering the cost basis of each resource. Expected behavior Minimal to no difference in memory pressure for the same workloads when switching between flytekit versions. Additional context to reproduce Fetch same data (in our case ~12Gb) using flytekit 1.4.2 and flytekit 1.5.0 and observe system memory usage. Screenshots Flytekit 1.4.2 Flytekit 1.5.0 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte