#3866 [Core feature] Parallelize Node Executions in FlytePropeller
Issue created by
hamersaw
Motivation: Why do you think this is important?
Currently, FlytePropeller uses a modified DFS to traverse the FlyteWorkflow DAG where each node evaluation is synchronous. This means that in certain scenarios DAG traversal can be a bottleneck. For example, extremely large workflows or tasks that operate synchronously.
Goal: What should the final outcome look like, ideally?
FlytePropeller should evaluate nodes in parallel. This could be done adding a layer of futures on top of the current traversal algorithm so that node scheduling remains consistent but the actual evaluation is done in the background.
Describe alternatives you've considered
Leaving it as is may result in poor performance in certain scenarios as described.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte