Hi everyone…
question about dynamic workflows.
have a @dynamic that reads from elasticsearch and get around 25k documents using scrolling to create chunks of 1000 documents to then call a process_chunk
The problem is I get OOMKilled in the @dynamic part of the code.
is possible to assign resources to a @dynamic like tasks?
c
cool-lifeguard-49380
07/26/2024, 6:01 PM
Yes you can assign resources like in a normal task but I would recommend not doing any heavy lifting in the dynamic itself. Use the dynamic to register a sub workflow with not predefined shape at runtime but do the heavy lifting in tasks.
f
freezing-airport-6809
07/27/2024, 1:15 AM
@dynamic is just a regular task you can also use task config