acoustic-carpenter-78188
10/16/2023, 11:01 PMfrom typing import Annotated, List
import pandas as pd
from sklearn.datasets import load_wine
from flytekit import HashMethod, dynamic, task, workflow
def hash_pandas_dataframe(df: pd.DataFrame) -> str:
return str(pd.util.hash_pandas_object(df))
CacheablePandasDataFrameT = Annotated[pd.DataFrame, HashMethod(hash_pandas_dataframe)]
@task(cache=True, cache_version="664fedb393489d052c874ddd2c9e7273907cd8ac")
def process_data(df: CacheablePandasDataFrameT) -> CacheablePandasDataFrameT:
return df.assign(target=lambda x: x["target"].where(x["target"] == 0, 1))
@dynamic(cache=True, cache_version="1.0")
def dynamic_workflow() -> List[CacheablePandasDataFrameT]:
df = load_wine(as_frame=True).frame
return [process_data(df=df)]
If you run dynamic_workflow in remote it will not cache the intermediate df even if the type-hint can be extrapolated and Flyte knows that the type is cacheable. Interestingly this deviates from local execution as locally the cache is hit correctly.
Expected behavior
Cache should be hit for successive executions assuming no changes in output of load_wine.
Reasons for the bug:
The difference is between how promises (also called bindings at runtime) and Literals are modeled.
Literals have a hash value field - here, on the other hand Bindings do not - here.
flytekit when creating a binding at runtime for a dynamic workflow, simply populates the scalar value and ignores the hash field - here, here, here and here
Solution:
Add a hash field to bindings and ensure that is populated in dynamic workflows.
We should also make flytekit always use a common method to transform either a promise to binding data or a literal to binding data
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte