We are having an issue with input parameters not being deterministic when moving up from flyte 1.13 ...
n
We are having an issue with input parameters not being deterministic when moving up from flyte 1.13 We are runnin pyflyte with --params params.yaml Seems to coincide with the msgpack update from 1.14. Do you no longer consider dicts to be deterministic input, even if resolved from the cli in flytekit/interaction/click_types.JsonParamType. Or is this a oversight @task(cache=Cache(version="hard-coded",serialize=True)) def generic_task_no_params(dataset: str) -> str: return dataset @task(cache=Cache(version="hard-coded",serialize=True)) def generic_task(dataset: str, params: dict) -> str: return dataset @workflow def generic_wf(dataset: str, params: dict) -> None: generic_task_no_params(dataset=dataset) # caching works generic_task(dataset=dataset, params=params) # no caching generic_task(dataset=dataset, params=params["key"]) # no caching Note that this worked perfect with cache hits when we were at 1.13 version
c
@nutritious-match-70820, I'm looking into this.
n
Any update on this? Do you need more information to verify the issue?
We fixed it by overriding the function flytekit.core.type_engine.DictTransformer.dict_to_binary_literal With https://github.com/flyteorg/flytekit/blob/master/flytekit/core/type_engine.py#L2063 which is deprecated. Seems like the new msgpack version doesn't support the usecase with nested yaml/json