Hi everyone… question about dynamic workflows. hav...
# flyte-support
r
Hi everyone… question about dynamic workflows. have a @dynamic that reads from elasticsearch and get around 25k documents using scrolling to create chunks of 1000 documents to then call a process_chunk The problem is I get OOMKilled in the @dynamic part of the code. is possible to assign resources to a @dynamic like tasks?
c
Yes you can assign resources like in a normal task but I would recommend not doing any heavy lifting in the dynamic itself. Use the dynamic to register a sub workflow with not predefined shape at runtime but do the heavy lifting in tasks.
f
@dynamic is just a regular task you can also use task config