https://flyte.org
Join Slack
Hey all, Currently seeing a flyte workflow issue, specifically: ```Failed to convert return value fo...
# flyte-support
b
Hey all, Currently seeing a flyte workflow issue, specifically:
Copy code
Failed to convert return value for var o0 for function src.tasks.batch_transform.load_bt_result with error <class 'TypeError'>: unhashable type: 'list'. 

SYSTEM ERROR! Contact platform administrators.
We've double checked that this task is actually returning a pandas df. We've bumped the cache version as well. The flyte propeller logs show:
Copy code
{"json":{"exec_id":"f6d3b8ca69b2149bbbbb","node":"n3/n3","ns":"forecaster-training-dev","res_ver":"38555329","routine":"worker-19","tasktype":"python-task","wf":"forecaster-training:dev:src.workflow.end_to_end"},"level":"warning","msg":"No plugin found for Handler-type [python-task], defaulting to [container]","ts":"2022-10-26T01:19:57Z"}
{"json":{"exec_id":"f6d3b8ca69b2149bbbbb","node":"n4/n3","ns":"forecaster-training-dev","res_ver":"38555329","routine":"worker-19","wf":"forecaster-training:dev:src.workflow.end_to_end"},"level":"warning","msg":"No plugin found for Handler-type [python-task], defaulting to [container]","ts":"2022-10-26T01:19:57Z"}
{"json":{"exec_id":"f6d3b8ca69b2149bbbbb","node":"n4/n3","ns":"forecaster-training-dev","res_ver":"38555329","routine":"worker-19","wf":"forecaster-training:dev:src.workflow.end_to_end"},"level":"warning","msg":"Failed to record taskEvent, error [EventAlreadyInTerminalStateError: conflicting events; destination: ABORTED, caused by [rpc error: code = FailedPrecondition desc = invalid phase change from FAILED to ABORTED for task execution {resource_type:TASK project:\"forecaster-training\" domain:\"dev\" name:\"src.tasks.batch_transform.load_bt_result\" version:\"KZ5UyzZ2adcfdjkL4edUwg==\"  node_id:\"n4-0-n3\" execution_id:\u003cproject:\"forecaster-training\" domain:\"dev\" name:\"f6d3b8ca69b2149bbbbb\" \u003e  0 {} [] 0}]]. Trying to record state: ABORTED. Ignoring this error!","ts":"2022-10-26T01:19:57Z"}
f
This is odd That the pandas dataframe is getting read as list
Task signature will help
b
Copy code
@task(cache=True, cache_version="1.7")
def load_bt_result(result: FlyteDirectory) -> CachedDataFrame:


CachedDataFrame = Annotated[
    pd.DataFrame, HashMethod(hash_pandas_dataframe_function)  # noqa: F821
]

def hash_pandas_dataframe_function(df: pd.DataFrame) -> str:
f
hmm interesting
is this because of custom hashing maybe? cc @high-accountant-32689 / @glamorous-carpet-83516?
h
@bumpy-morning-40916, can you double-check which versions of python, flytekit, flytepropeller, and flyteadmin you're running? Also, I noticed that the cache version in the task you shared is 1.7, does this mean that this used to work and broke recently?
b
Yes, was working previously but it was a simple fix... a recent change to the pandas df hashing function wasnt included in this repo causing the issue. thanks @freezing-airport-6809 & @high-accountant-32689 for the help
f
Ohh but that error was odd
166 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by