Hey team! I have a question related to flyte propeller’s latency. In the docs it says:

The worst case for FlytePropeller is workflows that have an extremely large fan-out. This is because FlytePropeller implements a greedy traversal algorithm…

If we have workflows with this characteristic (for example, one workflow spans 2k dynamic tasks), is there a recommendation on how to scale? Otherwise we get alerts due to latency (which make sense for this specific wf), but it’s hard to isolate this scenario from other ones. For what I see in the docs, the suggestion is to limit the amount of parallel tasks

Is there a problem or alert not great?

I want to understand if there is something to be tuned from the propeller side to reduce the latency for workflows with a huge number of dynamic tasks

Reduce max parallelism

That will limit the amount of dynamic tasks in parallel, right?

Yup

I understand the nature of the problem and I don’t think it’s wrong at all 🙂 . Just wanted to know this

@Franco Bocci there is a solution, Dan is working on true parallel evaluation, in the engine. Would you be open to help testing