@ancient-wolf-19325 I have a workflow containing two tasks, one reads a csv into dataframe and the other returns data from the dataframe. The workflow works for a small csv but fails if the csv is large enough with a

Expected checksum 96ZInw== did not match calculated checksum: GBZQVA==

Any way to solve this issue?

The error you're encountering is likely due to a mismatch in the expected and calculated checksums, which can occur when handling large data files. This issue might be related to the way Flyte handles large data transfers and the limitations on data size. To address this, consider the following steps: 1. Increase Data Size Limits: Check if there are any data size limits configured in your Flyte setup. You might need to increase the

max-output-size-bytes

or similar configuration settings to accommodate larger files. According to this discussion, adjusting the

max-output-size-bytes

can help resolve issues related to large data transfers. 2. Use FlyteFile for Large Data: Instead of passing large data directly between tasks, consider using

FlyteFile

to handle large files. This approach involves storing the file in a remote location (e.g., S3) and passing the file path between tasks, which can help avoid checksum issues. Would you like more details on how to implement these solutions? Sources: - this discussion 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.