New RunLLM response generated: >>>*# Question* Hello - we've been experimenting with using the ArrayNode

map_task

in 1.13 for very large jobs (>60,000 tasks) and have found an interesting failure case. It seems like once the job size crosses a certain threshold, we consistently see failures on the same

map_task

batch. The error is a generic # Answer The issue you're encountering with

map_task

for very large jobs is likely related to the limitations and known bugs in handling large-scale parallel tasks in Flyte. Specifically, Flyte has a configuration limit on the number of tasks that can be handled in a single

map_task

, which is typically set to 5,000 tasks due to state storage size restrictions in etcd (source). Additionally, there are known issues with

map_task

and

ArrayNode

handling in Flyte, such as serialization problems and external modifications to Kubernetes pods causing failures (source, source). These issues can manifest as generic errors when the system is under heavy load or when the task count exceeds the recommended limits. Would you like more detailed information on potential workarounds or configuration adjustments to handle large-scale

map_task

executions? Sources: - Flyte discussion on large task execution - Flyte issue on ArrayNode handling - Flyte issue on pod modifications Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1722545387.101249 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.