I've started seeing this error in propeller after upgrading flytekit to 1.14. Any setting available to resolve? Failed storing workflow to the store, reason: workflow too large

The error "workflow too large" indicates that the workflow size exceeds the storage limits in Flyte. This can occur due to large workflow states being stored in etcd. To resolve this, you can enable the

useOffloadedWorkflowClosure

setting, which offloads parts of the workflow to external storage, reducing the size stored in etcd. This setting can be configured in the FlyteAdmin configuration. Would you like more details on how to configure this setting? Sources: - Flyte Offloading Configuration 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 That didn't seem to fix the error. This is the message within the flytepropeller logs: "Failed storing workflow to the store, reason: workflow too large

If enabling

useOffloadedWorkflowClosure

did not resolve the issue, it might be necessary to further investigate the size of the workflow and consider additional strategies. Here are a few steps you can take: 1. Check Workflow Size: Ensure that the workflow size is indeed large and identify which parts are contributing to the size. This can include large inputs, outputs, or a high number of nodes. 2. Optimize Workflow: Consider breaking down the workflow into smaller sub-workflows or using dynamic workflows to manage large fan-outs. 3. Increase Storage Limits: If possible, adjust the storage limits in your Flyte setup. This might involve increasing the etcd size limits or configuring Flyte to handle larger payloads. 4. Review Configuration: Double-check that the

useOffloadedWorkflowClosure

setting is correctly configured and that both FlyteAdmin and FlytePropeller have access to the same blobstore. Would you like more detailed guidance on any of these steps? Sources: - Flyte Offloading Configuration - Flyte Workflow Size Issues 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.