Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Is there a hard limit on allowed for cache lookup? We are running into this error.
```array size &gt; max allowed for cache lookup. requested [15434]. writer allowed [10000] reader allowed [10000]```

The answer was it was a simple config change found here.
<https://docs.flyte.org/en/latest/deployment/cluster_config/flytepropeller_config.html#catalogcache-catalog-config>

a concern to think about is memory on the propellor

Yes, I would also recommend running sharded propeller