New RunLLM response generated: >>>*# Question* Hello everyone! We've been scaling up our workflows recently and running into an ongoing issue with memory usage in the flyte-binary (v1.13.3), so I've been investigating. As the workflows run, the memory of the flyte-binary pod steadily increases, sometimes exceeding its limits and crashing. I think this is expected, and we can try to increase the memory available to mitigate the crashes. However, I noticed that when the workflow finishes, it looks like the memory isn't released, which means no matter how much memory we allocate to the pod, it will eventually crash. If anyone has any workarounds or fixes for this I'd be grateful. # Answer This issue may be related to a memory leak in the Flyte system. According to this discussion, users have reported similar memory utilization issues with the flyte-binary pod, where memory usage increases over time and is not released, leading to pod eviction. This behavior suggests a potential memory leak. To address this, you can try the following: 1. Increase Memory Limits: Temporarily increase the memory limits for the flyte-binary pod to prevent immediate crashes. 2. Monitor and Debug: Use monitoring tools to track memory usage patterns and identify specific workflows or tasks that may be contributing to the issue. 3. Report and Collaborate: If the issue persists, consider reporting it to the Flyte community or checking for existing issues on GitHub to collaborate on a solution. Would you like more detailed steps on how to implement these changes or further troubleshooting steps? Sources: - Flyte Community Discussion Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732876317.258609 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.