Hi all, in the case that a task cache may take a long time to load, what are some possible ways to diagnose the issue? I'm trying to investigate why a task within a subworkflow took close to an hour to load from cache. I understand sometime can be due to image pulling which is normal, but an hour seems excessive, and its only a transient issue.

could you elaborate a bit please? what do you mean by cache?

I'm referring to the task output cache -

@task(cache=True, cache_version="x")

. If a tasks output is cached from a prior execution, and on re-run of the workflow which contains the cached task, then what could cause those outputs to take a long time to reload? Basically a user of ours had re-run an execution and with task caching turned on, the task took an hour load from cache.

that doesn’t sound… possible. how are you getting the times?

so something like this periodically happens, and I'm not sure if I can always attribute this to network or very large outputs. Have you observed anything like this happening before?

i have not seen this before.