@ancient-wolf-19325 what's the difference between @dynamic and @task nesting ?

It seems likely that the main difference is that

@dynamic

creates a dynamic workflow whose DAG is constructed at runtime, allowing you to use Python control flow (like loops and conditionals) to generate tasks and dependencies dynamically, while

@task

nesting (calling a task from within another task) is not supported and can lead to unexpected behavior—nested tasks run in the same process, not as separate Flyte tasks, and Flyte emits warnings or errors for this pattern. Dynamic workflows are the recommended way to achieve dynamic or conditional task execution in Flyte, as they are compiled and executed at runtime and can generate subworkflows on the fly, whereas tasks should not invoke other tasks directly except via workflows or dynamic workflows. For example, using

@dynamic

allows you to loop over inputs and spawn tasks dynamically, and each task runs as a separate Flyte execution unit. In contrast, calling a task from within another task just executes the function locally, not as a Flyte task, and is discouraged in the documentation and codebase. See Dynamic workflows and Nested @task invocations should not supported for more details. Would you like more detail or code examples? Sources: - Dynamic workflows - Nested @task invocations should not supported 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

can I set the cpu and memory request/limit on a dynamic @ancient-wolf-19325

Yes, you can set CPU and memory requests/limits on a

@dynamic

workflow using the

requests

and

limits

parameters, just like with

@task

. These resource specifications will apply to the dynamic workflow's execution container, which constructs the subworkflow at runtime. Use the

Resources

object to specify these values, e.g.,

@dynamic(requests=Resources(cpu="2", mem="4Gi"), limits=Resources(cpu="4", mem="8Gi"))

see API reference. Would you like a code example or more detail? Sources: - flytekit.core.environment dynamic API reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

is it expected that I can only see the container's log after the pod status is completed @ancient-wolf-19325?

No, it is not expected—you should be able to see a container's log while the pod is still running, not only after completion. Flyte provides log links (e.g., to Kubernetes, Cloudwatch, Stackdriver) that are available during task execution phases such as running or initializing, depending on plugin and configuration. Log links are typically shown as soon as the pod is created and the container is running, not just after completion, as described in the debugging executions guide and log link configuration docs. If you only see logs after completion, it may be due to log aggregator delays, plugin configuration, or cluster-specific issues. Would you like troubleshooting steps or more detail? Sources: - debugging executions guide - log link configuration docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

I checked the log directly on the pod but didn't see anything, and I'm pretty sure I should have some logs @ancient-wolf-19325

If you checked the pod logs directly (e.g., with

kubectl logs

) and see no output, but expect logs, this usually indicates that your task code is not writing to stdout/stderr, or the process may not have started, exited early, or is writing logs to a file instead of the console. Also, if the pod is still pending or initializing, logs may not be available yet. See debugging executions guide for steps to investigate. Would you like more detailed troubleshooting steps for missing pod logs? Sources: - debugging executions guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

does calling @dynamic recursively takes more memory @ancient-wolf-19325?

Yes, recursively calling

@dynamic

workflows can increase memory usage, as each dynamic invocation generates a new workflow specification (DAG) and stores metadata for all nodes, which accumulates with recursion depth and breadth. Flyte imposes limits on recursion depth to prevent excessive resource consumption and potential system instability, as described in the dynamic workflows documentation and merge sort example. Would you like more detail on how memory usage grows or best practices for recursive dynamic workflows? Sources: - dynamic workflows documentation - merge sort example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

so the next pod memory won't start from 0 @ancient-wolf-19325?

My understanding is that each new pod created for a dynamic workflow invocation starts with a fresh memory space—memory is not shared or carried over between pods. However, the overall memory usage at the system level can increase if many pods are running concurrently due to recursion, since each pod consumes its own resources. This is discussed in the dynamic workflows documentation. Would you like more detail on pod memory isolation or resource management in Flyte? Sources: - dynamic workflows documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.