blue-eve-64591
05/31/2022, 6:13 AMdef hash_pandas_dataframe(df: pandas.DataFrame) -> str:
return str(pandas.util.hash_pandas_object(df))
@task
def uncached_data_reading_task() -> Annotated[
pandas.DataFrame, HashMethod(hash_pandas_dataframe)
]:
return pandas.DataFrame({"column_1": [1, 2, 3]})
@task(cache=True, cache_version="1.0")
def cached_data_processing_task(df: pandas.DataFrame) -> pandas.DataFrame:
time.sleep(50)
return df * 2
@task
def compare_dataframes(df1: pandas.DataFrame, df2: pandas.DataFrame):
assert df1.equals(df2)
@workflow
def cached_dataframe_wf():
raw_data = uncached_data_reading_task()
# We execute `cached_data_processing_task` twice, but we force those
# two executions to happen serially to demonstrate how the second run
# hits the cache.
t1_node = create_node(cached_data_processing_task, df=raw_data)
t2_node = create_node(cached_data_processing_task, df=raw_data)
t1_node >> t2_node
# Confirm that the dataframes actually match
compare_dataframes(df1=t1_node.o0, df2=t2_node.o0)
if __name__ == "__main__":
df1 = cached_dataframe_wf()
print(f"Running cached_dataframe_wf once : {df1}")
but sometimes the caching works and sometimes it doesnt. we've tried running with pyflyte run --remote caching.py cached_dataframe_wf as well as trying the relaunch button but as you can see in the pictures it tends to not work and i'm not sure why. any ideas?icy-agent-73298
05/31/2022, 8:14 AMget_success_count
You can portforward your datacatalog pod similar to this
kubectl port-forward datacatalog-6797ff48c6-tvkm5 -n flyte 10254:10254
And access the metrics locally http://localhost:10254/metrics
Every cache hit will increment this counter . Also the UI shows the cache symbolicy-agent-73298
05/31/2022, 8:14 AMMaxCacheAge config.Duration `json:"max-cache-age" pflag:", Cache entries past this age will incur cache miss. 0 means cache never expires"`
icy-agent-73298
05/31/2022, 8:18 AMk logs -n flyte flytepropeller-6844db64cf-5jtxn |grep "Catalog CacheHit" |wc -l
blue-eve-64591
05/31/2022, 8:18 AMicy-agent-73298
05/31/2022, 8:24 AMicy-agent-73298
05/31/2022, 8:24 AMblue-eve-64591
05/31/2022, 8:27 AMblue-eve-64591
05/31/2022, 8:28 AMicy-agent-73298
05/31/2022, 8:29 AMblue-eve-64591
05/31/2022, 8:29 AMblue-eve-64591
05/31/2022, 8:47 AMblue-eve-64591
05/31/2022, 8:47 AMicy-agent-73298
05/31/2022, 11:27 AMblue-eve-64591
05/31/2022, 11:28 AMicy-agent-73298
05/31/2022, 11:28 AMicy-agent-73298
05/31/2022, 11:44 AMblue-eve-64591
05/31/2022, 11:45 AMblue-eve-64591
05/31/2022, 11:45 AMicy-agent-73298
05/31/2022, 11:46 AMblue-eve-64591
05/31/2022, 11:48 AMicy-agent-73298
05/31/2022, 11:49 AMblue-eve-64591
05/31/2022, 11:50 AMglamorous-carpet-83516
05/31/2022, 3:47 PMflytectl demo start --image pingsutw/sandbox-lite-test
freezing-airport-6809
blue-eve-64591
06/02/2022, 3:16 AMglamorous-carpet-83516
06/02/2022, 4:23 AM