hi folks, do you run Flyte -- local mode -- on notebook on transient notebook env (eg. databricks, collab)? In that situation, how do you enable local cache? the default behavior writes output to

~/.*flyte*/*local*-*cache*/

which assumes strongly a durable persistent local env

Wdym

right.. but if i run Flyte code on databricks environement for example.. cache just doens't work across runs

Wdym is there no disk

yah i see that ``~/.*flyte*/*local*-*cache*/` now has sq-lite artifact

Shall we collaborate on this

and flytekit likely queries on this local db

Ya it does

so if we implement s3 path, it's gonna look more like Flyte running on K8S cluster

No it won’t

just s3 look-up by uri path?

But if this is a custom cache then we could Simply upload the cache db

ok i'd love to collaborate

Ok I have never used it so would love to understand

databricks notebook have great UX and it's been worth the $$ for the productivity gain for our engs

@Mick Jermsurawong using remote cache locally is dangerous

it is dangerous in the sense that you are concerned about data corruption right?

Let me discuss today

hi ketan! any further thoughts on this?

i read it briefly, i have some comments, i guess i think if we set a s3 path you do not even need a prefix / context

if s3 path can be env var as well, that would work.. i'm happy to implement the work here, but want to make sure that directinoally it's something OSS will accept

yes i think we should, @Yee is out but he will be back week after (he got married)

sounds good. will work with Yee on this then

awesome, let's keep chatting here. It wouldn't be too hard to lean on our flytekit's existing infra to support loading/writing to a blob store. I just wanted to separate the two ideas: (1) the local scope, and (2) a remote cache. If you want to throw a PR I'd be more than happy to review.

gotcha.. 1/ local scope here is simply have the cache local disk to be configurable right? 2/ remote cache will also reuse that cache path? 2.1/ do you have preference if we will simply sync the whole DB files that python diskcache write.. or should we try to encode the cache key in indiviual remote blob store path (closer to how on-cluster execution works)

1/ correct. 2/ Yeah, the local scope is optional, its purpose is just to help you segregate local caches. 2.1/ That's a good question. It'd be simpler to sync all DB files, but I fear that this might make the local cache very slow after multiple runs (imagine the case of a few thousand objects of different sizes being stored there). I also dislike the fact that if we go that route the local cache becomes slower and slower... so my vote goes for to make each entry its own separate entry in the blob store. wdyt?

Sorry eduardo for late response. And happy new year! 2.1/ Yup each cache entry can have its own entry in the blob store. That makes sense to me