jolly-florist-66728
09/26/2024, 4:07 PMdata = {
'column1': np.random.randint(0, 100, size=10),
'column2': np.random.rand(10),
'column3': np.random.choice(['A', 'B', 'C', 'D'], size=10)
}
df = pd.DataFrame(data)
inputs = {
"data_source_name": "dashboards-test",
"data_source_description": "test",
"df_input": df
}
workflow_name = (
"workflowname"
)
remote = FlyteRemote(config=Config.auto())
workflow = remote.fetch_workflow(
project="project",
domain="production",
name=workflow_name,
)
execution = remote.execute(workflow, inputs=inputs)
execution = remote.wait(execution)
But it fails on this line remote.execute(workflow, inputs=inputs) with the error:
ValueError: Error encountered while executing 'load_data':
Failed to find a handler for <class
'flytekit.types.structured.structured_dataset.StructuredDataset'>, protocol
[flyte], fmt ['']
I am assuming that they problem is related to the dataframe but in the workflow's code I saw that it is also being passed a dataframe without any problem.
Any suggestion?
# Answer
The error you're encountering is due to the pd.DataFrame not being correctly wrapped as a StructuredDataset when using FlyteRemote.execute. According to this discussion, you need to wrap the dataframe with StructuredDataset before passing it as an input.
Here's how you can modify your code:
from flytekit import FlyteRemote, Config
from flytekit.types.structured import StructuredDataset
import pandas as pd
import numpy as np
data = {
'column1': np.random.randint(0, 100, size=10),
'column2': np.random.rand(10),
'column3': np.random.choice(['A', 'B', 'C', 'D'], size=10)
}
df = pd.DataFrame(data)
inputs = {
"data_source_name": "dashboards-test",
"data_source_description": "test",
"df_input": StructuredDataset(dataframe=df)
}
workflow_name = "workflowname"
remote = FlyteRemote(config=Config.auto())
workflow = remote.fetch_workflow(
project="project",
domain="production",
name=workflow_name,
)
execution = remote.execute(workflow, inputs=inputs)
execution = remote.wait(execution)
This should resolve the issue by ensuring the dataframe is correctly wrapped as a StructuredDataset.
Would you like more detail on this?
Sources:
- Hi all quick question what is the correct way to pass a raw
Slack Link
https://.slack.com/archives/CP2HDHKE1/p1727366866.522569 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.