#3842 [BUG] Dynamic workflows break caching of artifacts offloaded to blob storage Issue created by fg91 Describe the bug Some type transformers offload artifacts to blob storage as follows (e.g. here):

def to_literal(...) -> Literal:
        local_path = ...
        # Save object to local path

        remote_path = ctx.file_access.get_random_remote_path(local_path)
        ctx.file_access.put_data(local_path, remote_path, is_multipart=False)
        # Return Literal containing remote_path

When objects of such types are passed to

@dynamic

workflows and then passed along to tasks called within the dynamic workflow, this behaviour always leads to cache misses. The reason is that in the dynamic workflow, the objects are deserialized and then again serialized to a different random remote path. Expected behavior There should not be cache misses in this situation. Additional context to reproduce Let us consider this example workflow:

import torch.nn as nn
from flytekit import task, workflow, dynamic


@task(cache=True, cache_version="0.1")
def train(model: nn.Module) -> nn.Module:
    print(f"Training model {model}")
    return model


@task(cache=True, cache_version="0.1")
def other_task(param: int) -> int:
    print(f"Doing something else with param {param}")
    return param


@dynamic(cache=True, cache_version="0.1")
def sub_wf(model: nn.Module, param: int) -> tuple[nn.Module, int]:
    other_task(param=param)
    train(model=model)

    return model, param


@task(cache=True, cache_version="0.1")
def create_model() -> nn.Module:
    return nn.Linear(1, 1)


@workflow
def wf(param: int = 1):
    model = create_model()
    sub_wf(model=model, param=param)

Screenshots The first execution with

param=1

results in cache puts for all tasks: Next, let's re-run the workflow but with `param=2`: It is expected that

other_task

has a cache miss since we changed

param

. However, since

create_model

had a cache hit, so should have

train

. Instead, one can observe that the output of

create_model

(retrieved from cache) ... ... is not the same is as the input to `train`: The reason is that the respective type transformer deserialized the artifact and serialized it again to a new random bucket path. I don't know how exactly this could be solved but the type engine should "somehow detect that it is being executed in a dynamic workflow" and should "simply forward the artifacts instead of serializing them to a different location". Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte