https://flyte.org logo
#ask-the-community
Title
# ask-the-community
f

Franco Bocci

08/11/2023, 10:27 AM
Hey team! I have a question related to flyte propeller’s latency. In the docs it says:
The worst case for FlytePropeller is workflows that have an extremely large fan-out. This is because FlytePropeller implements a greedy traversal algorithm…
If we have workflows with this characteristic (for example, one workflow spans 2k dynamic tasks), is there a recommendation on how to scale? Otherwise we get alerts due to latency (which make sense for this specific wf), but it’s hard to isolate this scenario from other ones. For what I see in the docs, the suggestion is to limit the amount of parallel tasks
k

Ketan (kumare3)

08/11/2023, 1:45 PM
Is there a problem or alert not great?
f

Franco Bocci

08/11/2023, 1:46 PM
I want to understand if there is something to be tuned from the propeller side to reduce the latency for workflows with a huge number of dynamic tasks
I don’t think there is a bug. I just see high latency for these workflows and I’d like to know if there’s anything I could do
k

Ketan (kumare3)

08/11/2023, 1:49 PM
Reduce max parallelism
Sadly nothing else
f

Franco Bocci

08/11/2023, 1:49 PM
That will limit the amount of dynamic tasks in parallel, right?
k

Ketan (kumare3)

08/11/2023, 1:50 PM
Yup
f

Franco Bocci

08/11/2023, 1:50 PM
I understand the nature of the problem and I don’t think it’s wrong at all 🙂 . Just wanted to know this
Okay, I can make a decision with this. Thanks!
k

Ketan (kumare3)

08/11/2023, 1:52 PM
@Franco Bocci there is a solution, Dan is working on true parallel evaluation, in the engine. Would you be open to help testing
Cc @Dan Rammer (hamersaw)
2 Views