Hey all,
Currently seeing a flyte workflow issue, ...
# ask-the-communitye
Edgar Trujillo
10/26/2022, 2:26 AMHey all,
Currently seeing a flyte workflow issue, specifically:
Copy code
Failed to convert return value for var o0 for function src.tasks.batch_transform.load_bt_result with error <class 'TypeError'>: unhashable type: 'list'.
SYSTEM ERROR! Contact platform administrators. Copy code
{"json":{"exec_id":"f6d3b8ca69b2149bbbbb","node":"n3/n3","ns":"forecaster-training-dev","res_ver":"38555329","routine":"worker-19","tasktype":"python-task","wf":"forecaster-training:dev:src.workflow.end_to_end"},"level":"warning","msg":"No plugin found for Handler-type [python-task], defaulting to [container]","ts":"2022-10-26T01:19:57Z"}
{"json":{"exec_id":"f6d3b8ca69b2149bbbbb","node":"n4/n3","ns":"forecaster-training-dev","res_ver":"38555329","routine":"worker-19","wf":"forecaster-training:dev:src.workflow.end_to_end"},"level":"warning","msg":"No plugin found for Handler-type [python-task], defaulting to [container]","ts":"2022-10-26T01:19:57Z"}
{"json":{"exec_id":"f6d3b8ca69b2149bbbbb","node":"n4/n3","ns":"forecaster-training-dev","res_ver":"38555329","routine":"worker-19","wf":"forecaster-training:dev:src.workflow.end_to_end"},"level":"warning","msg":"Failed to record taskEvent, error [EventAlreadyInTerminalStateError: conflicting events; destination: ABORTED, caused by [rpc error: code = FailedPrecondition desc = invalid phase change from FAILED to ABORTED for task execution {resource_type:TASK project:\"forecaster-training\" domain:\"dev\" name:\"src.tasks.batch_transform.load_bt_result\" version:\"KZ5UyzZ2adcfdjkL4edUwg==\" node_id:\"n4-0-n3\" execution_id:\u003cproject:\"forecaster-training\" domain:\"dev\" name:\"f6d3b8ca69b2149bbbbb\" \u003e 0 {} [] 0}]]. Trying to record state: ABORTED. Ignoring this error!","ts":"2022-10-26T01:19:57Z"}k
Ketan (kumare3)
10/26/2022, 3:08 AM
This is odd
That the pandas dataframe is getting read as list
Ketan (kumare3)
10/26/2022, 3:08 AM
Task signature will help
e
Edgar Trujillo
10/26/2022, 3:10 AM Copy code
@task(cache=True, cache_version="1.7")
def load_bt_result(result: FlyteDirectory) -> CachedDataFrame:
CachedDataFrame = Annotated[
pd.DataFrame, HashMethod(hash_pandas_dataframe_function) # noqa: F821
]
def hash_pandas_dataframe_function(df: pd.DataFrame) -> str:k
Ketan (kumare3)
10/26/2022, 3:47 AM
hmm interesting
Ketan (kumare3)
10/26/2022, 3:47 AM
is this because of custom hashing maybe? cc @Eduardo Apolinario (eapolinario) / @Kevin Su?
e
Eduardo Apolinario (eapolinario)
10/26/2022, 4:19 AM@Edgar Trujillo, can you double-check which versions of python, flytekit, flytepropeller, and flyteadmin you're running? Also, I noticed that the cache version in the task you shared is 1.7, does this mean that this used to work and broke recently?
e
Edgar Trujillo
10/26/2022, 12:51 PMYes, was working previously but it was a simple fix... a recent change to the pandas df hashing function wasnt included in this repo causing the issue. thanks @Ketan (kumare3) & @Eduardo Apolinario (eapolinario) for the help
k
Ketan (kumare3)
10/26/2022, 2:33 PM
Ohh but that error was odd
164 Views