Lee Ning Jie Leon10/10/2023, 9:32 AM
k top pod NAME CPU(cores) MEMORY(bytes) flytepropeller-ddb88df5-4wltg 6m 758Mi flytepropeller-ddb88df5-j8z4c 2m 67Mi
Lee Ning Jie Leon10/11/2023, 10:11 AM
Did you run propeller managerno, didnt know we have to 😅 , assumed it will be load balanced once we have replica > 1 like other components i'll test out the sharding, thanks for referring to the docs.
Also you should tweak some configs. Single propeller can handle 1000s of workflows per secondi checked the metrics but they don't show any anomaly
We probably have at most 20 workflows running at any time for now on a 1Gi mem propeller. We'll just increase the mem and monitor further. Thanks!
flyte:propeller:all:free_workers_count sum(rate(flyte:propeller:all:round:raw_ms[5m])) by (wf) sum(rate(flyte:propeller:all:main_depth[5m]))
• it seem that the memory utilisation increases with time, are there any internal caching by the propeller that cause this behaviour?with regards to this, i reduced the number of replica from 2 to 1, since the other propeller isn't in used anyway So the resource that the single replica is doubled, but the memory consumption drop drastically. The number of running workflow after deployment was even higher
Lee Ning Jie Leon10/13/2023, 4:04 AM