Hey all, sometimes the metadata cache isnt found and we are getting this error (v1.13.5)

Workflow[flytesnacks:development:brainflyte.features.window_metrics.wf_get_window_metrics.calculate_window_metrics_and_dump_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: Failed to check Catalog for previous results: failed to read inputs when trying to query catalog: [READ_FAILED] failed to read data from dataDir [<s3://bs-flyte-prod/metadata/propeller/flytesnacks-development-a4vflgwx794sbt8srzqw/n0/data/0/dn422/0/prefilterandcrop/inputs.pb>]., caused by: path:<s3://bs-flyte-prod/metadata/propeller/flytesnacks-development-a4vflgwx794sbt8srzqw/n0/data/0/dn422/0/prefilterandcrop/inputs.pb>: not found

The weird thing is that the current execution id is a4vflgwx794sbt8srzqw (not a previous one), so it doesnt find inputs from this same execution? a little odd This is so unexpected and makes it hard to use, because we have to update the cache version for it to work again Anybody got a lead? maybe the metadata is deleted somehow?

@best-oil-18906, does the file exist in s3? Also, can you share the "shape" of your workflow? For example, does it contain @dynamic , sub-workflows, etc. In the case of ​`prefilterandcrop`​, can you share the signature and how you're invoking it?