#4580 [Core feature] Remote cache for local execution in ephemeral environment Issue created by mickjermsurawong-stripe ### Motivation: Why do you think this is important? • Caching is essential feature of Flyte execution • Running Flyte local execution on notebook works well in with durable file system, as content is written to this fixed local path • However, when running so on notebook with ephemeral file system, the written content goes away across different notebook sessions. Local caching then is no-op is this environment. ### Goal: What should the final outcome look like, ideally? Running Flyte local execution in ephemeral filesystem should support caching. Essentially, this local execution in ephemeral environment is similar to the stateless remote execution on K8S where caching relies on external storage. We want to similarly enable remote cache for this ephemeral local execution. ### Describe alternatives you've considered Configuring the ephemeral environment to have persistent storage across notebooking sessions is possible. But for every such environment, we have to implement a bespoke solution. ### Propose: Link/Inline OR Additional context Plumbing the info One problem here is defining the scope of cache key. In remote execution, we have domain/project. For such local execution, client should be able to provide its own ID that determines scope of caching. It's already the intention to make cache path configurable. We can further add the scope of cache key here as well. So the config may look like:

local_cache_path: <local FS path>

or

external_local_cache_path: <s3://...>
local_cache_scope: <...>

Ideally we can also dynamically set local cache scope via env var for convenience.

LOCAL_CACHE_SCOPE

Cache implementation • Option 1/ use the local cache scope as part of the cache key and do lookup on S3 • Option 2/ rely on the existing

diskcache

impl that excludes cache scope, but upload the entire db artifacts to S3 with the cache scope key...

<s3://my-external-local-cache/<cache_scope>/cache.db>
<s3://my-external-local-cache/<cache_scope>/cache.db-shm>
<s3://my-external-local-cache/<cache_scope>/cache.db-wal>

### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte