We've encountered some interesting production issues recently. We ramped up the number of workflows running concurrently in Flyte and we discovered that when those workflows included dynamic tasks our CPU usage dramatically increased. After doing some profiling it seems that 25% or more of the CPU time of propeller is involved in just deserializing dynamic workflow CRDs from the blob store. We found that this also introduced memory pressure on the system with frequent GC, presumably due to the ephemeral objects constantly getting created and thrown away. We found that you can enable an in-memory cache for the dynamic workflow CRDs but it doesn't really help because the cache is still binary so you still incur the cost of JSON deserialization on every workflow evaluation. We're considering building an in-memory LRU cache to avoid the repeated/duplicated deserializations but curious if anyone else has seen issues like this with dynamic workflows. I'm also wondering if we can just leverage sub launch plans as well since our etcD is not under much load at all.

There is a deserialized protobuf cache right? or you can use memory pools

This is the dynamic task workflow CRD so it’s JSON. There are in-memory byte array caches that I found backed by freecache. Didn’t really like the implementation and how you have to tune/hack GC to make the rest of go GC work correctly. Anyway, even with the in memory byte array cache you still do the cost of deserialization which is where the CPU is getting wrecked.

What is a dynamic workflow CRD?

buildContextualDynamicWorkflow: https://github.com/flyteorg/flyte/blob/master/flytepropeller/pkg/controller/nodes/dynamic/dynamic_workflow.go#L130-L222 Which calls FutureFIleReader: https://github.com/flyteorg/flyte/blob/71b8f72d2304ff92a13acd84196e600b82faff06/flytepropeller/pkg/controller/nodes/task/future_file_reader.go#L[…]7 And this is pulling from the blob store, and deserializing binary into the workflow CRD