Hi Team, what’s the recommended resource limits for flytepropeller?
We’re seeing it OOM even with 4Gb and just a couple of hundred workflows, which is strange.
10/11/2023, 12:54 PM
Yes this is strange
Is this single binary
What’s your cache config
We rarely if ever have seen ooms - points to a weird config
10/11/2023, 1:10 PM
Hey Ketan, I work with @Joe Hartshorn, we're pretty new to Flyte.
It's not single binary, we're deploying with the flyte-core chart, basically unchanged from stock config-wise. We haven't looked at cache config at all yet.
are you running hundred workflows in the same time? how many tasks per workflow
10/12/2023, 7:05 AM
It was 365x 1 workflow with 3 tasks. It seems to have calmed down and it might be because:
In a previous version of the task I was passing a VERY long list as an output/input, now the tasks pass a data frame. If the list wasn’t offloaded to the data plane it might have needed to be loaded in memory for each execution. I can see how that would have make flytepropeller unhappy