New RunLLM response generated: >>>*# Question* Perhaps related: I'm testing fan-out using

map_task

with progressively larger batches. In the graphs below, the first ramp is ~1,000 elements, the second is ~10,000, and the current is ~20,000. The task is straightforward: It sleeps between 0.25 and 0.5 seconds and returns a dataclass. The memory usage seems disproportionate with the job. # Answer The memory usage issue you're experiencing with

map_task

could be related to how Flyte handles task execution and state storage. According to this discussion, Flyte uses etcd to store workflow execution state, and there are limits on object sizes in etcd (e.g., 1.5MB on EKS). This can lead to increased memory usage when handling large numbers of tasks. Additionally, this GitHub issue highlights that Flytekit 1.5.0 uses more RAM compared to Flytekit 1.4.2, potentially due to the way fsspec handles memory for file operations. This could be contributing to the memory usage you're observing. Would you like more details on potential solutions or optimizations for handling large fan-out tasks in Flyte? Sources: - Flyte Community Discussion - Flyte GitHub Issue #3665 Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1723822118.879969 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.