Hi all I m running into an issue with task caching on a Flyt

Question

Hi all I m running into an issue with task caching on a Flyte workflow using a container image where the `diskcache` package seems to be unable to create the relevant local cache directory is this expected is there something further I need to do to enable caching I ve registered this workflow and I m running it on the Flyte app not on my local machine The workflow runs fine when setting `cache=False` but I d love to be able to take advantage of Flyte s caching here to avoid rerunning on the same files repeatedly ```IMAGE SPEC = ImageSpec env= redacted pip index= redacted packages= redacted base image= <http ghcr io flyteorg flytekit py3 11 1 8 3|ghcr io flyteorg flytekit py3 11 1 8 3> registry= redacted task container image=IMAGE SPEC cache=True cache version= 0 1 def download file document id int gt Document ``` ```Traceback most recent call last File usr local lib python3 11 site packages flytekit exceptions scopes py line 206 in user entry point return wrapped args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^ File root data download py line 86 in process documents doc = download file document id=document ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages flytekit core base task py line 304 in call return flyte entity call handler self args kwargs type ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages flytekit core promise py line 1022 in flyte entity call handler result = cast LocallyExecutable entity local execute child ctx kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages flytekit core base task py line 268 in local execute outputs literal map = LocalTaskCache get self name self metadata cache version input literal map ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages flytekit core local cache py line 66 in get LocalTaskCache initialize File usr local lib python3 11 site packages flytekit core local cache py line 54 in initialize LocalTaskCache cache = Cache CACHE LOCATION ^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages diskcache core py line 450 in init raise EnvironmentError Message Errno 13 Cache directory home flytekit flyte local cache does not exist and could not be created User error ```

Eduardo Apolinario (eapolinario) · Answer

< Joe Kelly> is `download file` being invoked inside another task

Joe Kelly · Answer

Yes it is ``` task container image=IMAGE SPEC def process documents documents typing List int gt typing List Document docs = for document in documents doc = download file document id=document docs append doc return docs```

Joe Kelly · Answer

I can restructure this as a dynamic workflow e g removing the middle layer of `process documents ` and performing the loop directly in a dynamic `download files ` but then my caching doesn t work as I would like it to In my restructured case the cache only avoids rerunning the download task if the full list of files is precisely the same the behavior that I want is for Flyte to recognize if some subset of the listed files are cached to ignore those and to only run on the new files in the list

Joe Kelly · Answer

Ah I think I figured out the correct way to do what I m doing here I ve kept my `download file ` to process a single file so I get the desired caching benefits but I ve removed the intermediate `process documents ` task call and I m now calling `download file` directly with a `MapTask` from my workflow ```files = map task download file document id=documents ``` This seems to be running and caching properly Thanks for pointing me in the right direction < Eduardo Apolinario eapolinario >

Eduardo Apolinario (eapolinario) · Answer

awesome Glad I could help you figure it out