<#4833 [Core feature] Compress workflow state when...
# flyte-github
c
#4833 [Core feature] Compress workflow state when storing it in etcd Issue created by Tom-Newton ### Motivation: Why do you think this is important? Same motivation as #4569 In my mind the bigger we can make our workflows before hitting etcd size limits the better. ### Goal: What should the final outcome look like, ideally? Currently the workflow state is stored as a JSON string in etcd. JSON is not a very efficient data format. One alternative might be to encode to protobuf instead of JSON. This would provide savings for all the field names and all the numeric values that are currently stored as strings. I know that map tasks already do some compression of their state for storing in etcd. I'm proposing expanding that idea and applying it more generally. ### Describe alternatives you've considered Map tasks: Map tasks already use a compressed form for all the workflow state which is great. However, they currently have some other limitation like being unable to launch sub-workflows or any other node types besides just individual tasks. Map tasks are also a less general solution. Split up the workflow into lots of separate launchplans: This works and we do currently use it but it makes things less ergonomic and it requires the end user to have knowledge of implementation details of Flyte to understand how this helps and how to use it most effectively.
useOffloadedWorkflowClosure
This avoids writing the spec field to etcd but the status field is much bigger. On dynamic tasks can easily be very small so this doesn't really help much. ### Propose: Link/Inline OR Additional context No response ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte