<#3866 [Core feature] Parallelize Node Executions ...
# flyte-github
a
#3866 [Core feature] Parallelize Node Executions in FlytePropeller Issue created by hamersaw Motivation: Why do you think this is important? Currently, FlytePropeller uses a modified DFS to traverse the FlyteWorkflow DAG where each node evaluation is synchronous. This means that in certain scenarios DAG traversal can be a bottleneck. For example, extremely large workflows or tasks that operate synchronously. Goal: What should the final outcome look like, ideally? FlytePropeller should evaluate nodes in parallel. This could be done adding a layer of futures on top of the current traversal algorithm so that node scheduling remains consistent but the actual evaluation is done in the background. Describe alternatives you've considered Leaving it as is may result in poor performance in certain scenarios as described. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte