Lee Ning Jie Leon
10/10/2023, 9:32 AMk top pod
NAME CPU(cores) MEMORY(bytes)
flytepropeller-ddb88df5-4wltg 6m 758Mi
flytepropeller-ddb88df5-j8z4c 2m 67Mi
Ketan (kumare3)
Lee Ning Jie Leon
10/11/2023, 10:11 AMDid you run propeller managerno, didnt know we have to 😅 , assumed it will be load balanced once we have replica > 1 like other components i'll test out the sharding, thanks for referring to the docs.
Also you should tweak some configs. Single propeller can handle 1000s of workflows per secondi checked the metrics but they don't show any anomaly
flyte:propeller:all:free_workers_count
sum(rate(flyte:propeller:all:round:raw_ms[5m])) by (wf)
sum(rate(flyte:propeller:all:main_depth[5m]))
We probably have at most 20 workflows running at any time for now on a 1Gi mem propeller. We'll just increase the mem and monitor further. Thanks!• it seem that the memory utilisation increases with time, are there any internal caching by the propeller that cause this behaviour?with regards to this, i reduced the number of replica from 2 to 1, since the other propeller isn't in used anyway So the resource that the single replica is doubled, but the memory consumption drop drastically. The number of running workflow after deployment was even higher
Ketan (kumare3)
Lee Ning Jie Leon
10/13/2023, 4:04 AMKetan (kumare3)