@ancient-wolf-19325 regarding parallelization, what do workflows in a chunk share, what is the impact on chunk size In terms of resources and flytepropeller https://docs.flyte.org/en/latest/user_guide/advanced_composition/nested_parallelization.html

In Flyte, workflows in a chunk share the same execution context, which includes resource allocation and scheduling constraints. The chunk size impacts how tasks are batched and executed in parallel, affecting resource utilization and execution efficiency. According to the Flyte documentation on nested parallelization, a larger chunk size can lead to better resource utilization and reduced overhead from task scheduling, but it may also increase the memory and CPU load on individual nodes. Conversely, a smaller chunk size can lead to more granular parallelism and potentially better load distribution across nodes, but it may introduce higher scheduling overhead. Would you like more details on how to configure chunk size or its impact on specific workflows? Sources: - Flyte documentation on nested parallelization 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.