Hey community, for people running the single flyte-binary, what is your experience with memory utilization? We're seeing our flyte-binary pod get killed about once a week due to ever-increasing memory utilization (we're running 1.8.1). As you can see on the attached graph, it is sometimes the case that there appears to be some kind of garbage-collection or similar and utilization drops dramatically before climbing again, but mostly we experience the pod getting evicted once a week as it gets beyond 5-6GB and runs out of memory on the 8GB node. Thoughts? Config options? Thanks! (the lines in the image are all the flyte-binary pod -- when you see one go up and not come down, it is killed, and the next line in a slightly different shade is a new flyte-binary pod getting scheduled to replace the evicted one)

hmm seems like a memory leak

very interesting. Definitely has the hallmarks of a memory leak. Will open an issue to track this.

Hey @high-accountant-32689, yes we are running on AWS. Our core services are provisioned with Karpenter and we use a nodeSelector to cause these services to land on smallish machines (8GB memory). We previously had no mem requests or limits for the flyte-binary, but have installed a request of ~5Gi per goldilocks suggestion. Our execution throughput is probably small in the flyte world - we execute 10s of workflows per day, each of which have a handful of python tasks. They typically are running signal-processing on lots of cores using considerable memory, and run for 10-30 minutes. Occasionally a workflow may be run that employ a map_task running 10s of tasks in parallel. We do employ dynamic workflows quite a lot, do dynamic resource alloc via

with_overrides

, and have recently been working to add caching wherever possible. As a developer I make use of the Flyte Console all day long to view executions, inputs/outputs, etc. I am quite often (many times a day) registering new versions of all workflows/tasks (~100 total) to my dev project/domain for testing on the k8s/flyte cluster via pyflyte to serialize and then flytectl to register against a new image, and this also happens on the production project/domain several times a week as part of CI/CD. (Our wheel-based distribution which employs compiled C extensions makes "fast registration" something we've not managed to use yet). Probably many of those details don't matter to flyte-binary operation, just giving as much context as possible.

Great, thanks for the details, @microscopic-furniture-57275. Tracking this investigation in https://github.com/flyteorg/flyte/issues/3991.

What are the settings / config - especially the storage config and cache?

@freezing-airport-6809 @high-accountant-32689 Here is the config for our flyte-binary as deployed via Pulumi. I am not super-familiar with flyte-binary configuration but am learning. :)

@microscopic-furniture-57275, thanks for the update. It's in our list of things to investigate and will be prioritized shortly. We have a few suspects (the first one being the prometheus integration), but couldn't confirm yet. We'll update this thread when we find out more.

Thanks, FWIW we are using DataDog, but I didn't do the integration so am not certain what is involved there.

And this is only in single binary?

I have evidence that’s caused by Prometheus metrics. We will get this sorted out next week.

@freezing-airport-6809 Yes, this is in single-binary. This is the only version we run, so I'm not sure if it's exclusively a single-binary problem or not.