Threw this issue up on the Flyte repo: https://github.com/flyteorg/flyte/issues/5936 We were having some ominous incomplete write errors with both built-in IO and custom, registered serde methods when it came to writing large files. This popped up on an incremental update of

fsspec

to 2024.10.0, and was resolved by pinning

fsspec

to 2024.9.0. While it is possible that this is some complex issue within our own environment, the only interaction we saw was with the combination of inter-task Flyte IO and the above

fsspec

version.