New RunLLM response generated: >>>*# Question* Hi Team. I observed a behaviour of flyte-binary pod. When the system was under heavy load. the memory consumption of the pod kept on increasing from 500 MB to 2 GB , after which it got OOM Killed. I am not sure if this is due to internal caching / dataCatalog or it loads all Workflows in memory. If all workflows are kept in memory, is there a feature to delete the completed workflows? # Answer The increasing memory consumption of the Flyte-binary pod under heavy load, leading to OOM kills, is a known issue. According to this GitHub issue, memory consumption can increase due to Prometheus metrics in FlytePropeller, which might contribute to the memory usage every time a task runs. Regarding the deletion of completed workflows, Flyte does have a mechanism for cleaning up old workflows. The default TTL (Time To Live) for workflows to be garbage collected can be configured in the FlytePropeller configuration. You can reduce the TTL to ensure that completed workflows are cleaned up more frequently. This is detailed in this discussion. Would you like more details on how to configure the TTL for workflow garbage collection? Sources: - this GitHub issue - this discussion Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1722259096.674179 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.