#3842 [BUG] Dynamic workflows break caching of artifacts offloaded to blob storage Issue created by fg91 ### Describe the bug Some type transformers offload artifacts to blob storage as follows (e.g. here): def to_literal(...) -> Literal: local_path = ... # Save object to local path remote_path = ctx.file_access.get_random_remote_path(local_path) ctx.file_access.put_data(local_path, remote_path, is_multipart=False) # Return Literal containing remote_path When objects of such types are passed to

@dynamic

workflows and then passed along to tasks called within the dynamic workflow, this behaviour always leads to cache misses. The reason is that in the dynamic workflow, the objects are deserialized and then again serialized to a different random remote path. ### Expected behavior There should not be cache misses in this situation. ### Additional context to reproduce Let us consider this example workflow: import torch.nn as nn from flytekit import task, workflow, dynamic @task(cache=True, cache_version="0.1") def train(model: nn.Module) -> nn.Module: print(f"Training model {model}") return model @task(cache=True, cache_version="0.1") def other_task(param: int) -> int: print(f"Doing something else with param {param}") return param @dynamic(cache=True, cache_version="0.1") def sub_wf(model: nn.Module, param: int) -> tuple[nn.Module, int]: other_task(param=param) train(model=model) return model, param @task(cache=True, cache_version="0.1") def create_model() -> nn.Module: return nn.Linear(1, 1) @workflow def wf(param: int = 1): model = create_model() sub_wf(model=model, param=param) ### Screenshots The first execution with

param=1

results in cache puts for all tasks: [Screenshot 2023-07-06 at 15 57 48](https://private-user-images.githubusercontent.com/36511035/251487490-50a7e5d7-996f-4be1-8905-71fbe578fd8f.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjM3NzA0NjUsIm5iZiI6MTc2Mzc3MDE2NSwicGF0aCI6Ii8zNjUxMTAzNS8yNTE0ODc0OTAtNTBhN2U1ZDctOTk2Zi00YmUxLTg5MDUtNzFmYmU1NzhmZDhmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTExMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMTIyVDAwMDkyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTMzZDM1YzA0NjE5NTFmODAyMjkyYjQ3MmFhMDQwMjQ1YWU2MzU4ODE2OWJlNDBiMDY0YWNjMmRiMzMzMzViZDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ZUToEyayYAVlkdF-KJteHXUpPhfo7_pz-bKIO2Xdgjc) Next, let's re-run the workflow but with `param=2`: [Screenshot 2023-07-06 at 16 09 30](https://private-user-images.githubusercontent.com/36511035/251487820-3b8b5b3e-c51a-4da2-9a86-6883323bf294.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjM3NzA0NjUsIm5iZiI6MTc2Mzc3MDE2NSwicGF0aCI6Ii8zNjUxMTAzNS8yNTE0ODc4MjAtM2I4YjViM2UtYzUxYS00ZGEyLTlhODYtNjg4MzMyM2JmMjk0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTExMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMTIyVDAwMDkyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVjODY4ZTVhYzI4YmFmMGY4MWMyMDc5YTVlY2EzMmY3NTY4ZTg4MzYwNzQzYTNhMzM2MDAyM2Y1ODY2Y2ZlYTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.z-USEpl7L38VkQ4DSRtYmeKpkSlJgZc_8InkCduFNTU) It is expected that

other_task

has a cache miss since we changed

param

. However, since

create_model

had a cache hit, so should have

train

. Instead, one can observe that the output of

create_model

(retrieved from cache) ... [Screenshot 2023-07-06 at 16 09 44](https://private-user-images.githubusercontent.com/36511035/251488363-6f838d18-a064-4f8a-94fc-a1091ef41363.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjM3NzA0NjUsIm5iZiI6MTc2Mzc3MDE2NSwicGF0aCI6Ii8zNjUxMTAzNS8yNTE0ODgzNjMtNmY4MzhkMTgtYTA2NC00ZjhhLTk0ZmMtYTEwOTFlZjQxMzYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTExMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMTIyVDAwMDkyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg2ODlkZWU2ZjEyNjBjN2QzZjdhNWU5MzI0MjUxYzJhMzkxNTQ2YTllMTE2ODA1NTJkZWFkYTNjZGVhOTE0ZDUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.BoK7k5sU1zn2vSLX0s-LWcG-XXQL49Eas1xAdOo6RjM) ... is not the same is as the input to `train`: <https://private-user-images.githubusercontent.com/36511035/251488469-44b0b004-8237-4983-b692-acff144a14c7.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjM3NzA0NjUsIm5iZiI6MTc2Mzc3MDE2NSwicGF0aCI6Ii8zNjUxMTAzNS8yNTE0ODg0NjktNDRiMGIwMDQtODIzNy00OTgzLWI2OTItYWNmZjE0NGExNGM3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTExMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMTIyVDAwMDkyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTYxMzZiNmYxNzNhZGRjMmM4NDMwOWU2MWM2Y2Y3MTJiNjEyNTE3OTZiYWJmOGExZGM1YmViYzQ0NWEwN2FhMGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.eZRmI3X3OPsVhZ3d7dtUZH8iAy-XvODCOfxZposqBig|[Screenshot 2023-07-06 at 16 09 55](https://private-user-images.githubusercontent.com/36511035/251488469-44b0b004-8237-4983-b692-acff144a14c7.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSI… flyteorg/flyte