New RunLLM response generated: >>>*# Question* Hi everyone… question about dynamic workflows. have a @dynamic that reads from elasticsearch and get around 25k documents using scrolling to create chunks of 1000 documents to then call a process_chunk The problem is I get OOMKilled in the @dynamic part of the code. is possible to assign resources to a @dynamic like tasks? # Answer Yes, you can assign resources to a

@dynamic

task in Flyte, similar to how you would for regular tasks. This can be done using the

with_overrides

method or by specifying the resource requirements directly in the

@dynamic

decorator. Here's an example of how to set resource requests and limits for a

@dynamic

task:

from flytekit import dynamic, task, Resources

@task
def process_chunk(chunk: List[dict]) -> str:
    # Process the chunk of documents
    return "Processed"

@dynamic(
    task_config=Pod(
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary",
                    resources=V1ResourceRequirements(
                        requests={"cpu": "2", "memory": "4Gi"},
                        limits={"cpu": "4", "memory": "8Gi"},
                    ),
                )
            ],
        ),
    )
)
def dynamic_workflow(chunks: List[List[dict]]) -> List[str]:
    results = []
    for chunk in chunks:
        result = process_chunk(chunk=chunk)
        results.append(result)
    return results

In this example, the

dynamic_workflow

is configured to request 2 CPUs and 4Gi of memory, with limits set to 4 CPUs and 8Gi of memory. Would you like more details on this or any other aspect of Flyte? Sources: - Flyte GitHub example - GitHub issue on dynamic workflow caching Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1722013710.210659 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.