<@U06RKAMK0UC> OOM killed when returning a BaseMod...
# ask-ai
m
@ancient-wolf-19325 OOM killed when returning a BaseModel object back to dynamic workflow. The task has requested 55G of memory but still got killed.
j
Reasoning Complete! šŸ¤– šŸ” Searched knowledge base: I'll look up information about why a task might be OOM killed even after requesting a large amount of memory, especially when returning a BaseModel object in a dynamic workflow, to find possible causes and solutions. • Searching over 10 data sources from the following: File, Github File, Github Issue, Web • Found 59 documents that could be relevant from the following data sources: Github File, Github Issue, Github Pull Request, Web Content. • Filtered down to the top 30 most relevant documents. āœļø Generated answer: I have found clear documentation explaining why this OOM issue happens and what you can do to work around it. I'm ready to provide a complete answer.
If your Flyte task is OOM killed when returning a Pydantic BaseModel object to a dynamic workflow, even with 55Gi memory requested, this may be due to serialization overhead or a bug in how Flyte handles large/complex objects in dynamic workflows. There are known issues with memory growth and OOMs in dynamic tasks, especially when iterating over or returning large objects, or when using complex types like Pydantic models or dictionaries as dynamic task outputs or inputs. This can cause excessive memory usage or even infinite loops, leading to OOM kills regardless of the memory requested [BUG] Infinite loop and memory growth in dynamic task, [BUG] nested dynamic won't bind pydantic models or dictionaries as inputs. A recommended workaround is to serialize the BaseModel to a JSON string before returning it from the task, and then deserialize it in the downstream dynamic workflow. This avoids Flyte's type system issues with complex objects in dynamic contexts. If you are already serializing and still see OOMs, the issue may be with the sheer size of the object or a bug in Flyte's dynamic workflow handling. Also, ensure you are not iterating over promises or large lists of files/objects in the dynamic task, as this can cause runaway memory usage [BUG] Infinite loop and memory growth in dynamic task. Would you like more detail or example code for the serialization workaround? Sources: - https://github.com/flyteorg/flyte/issues/5164 - https://github.com/flyteorg/flyte/issues/5473 šŸ‘ 0 button šŸ‘Ž 0 button šŸ’” Hint: Mention @runllm in the thread for followups.
n
@microscopic-needle-82602 could this OOM Killed be about Ephemeral storage and not RAM ?