Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
m
Mohd Shahid Khan Afridi
02/27/2023, 1:27 PMI am trying to run basic BigQueryTask example provided in bq plugin doc, when it runs the
convert_bq_table_to_pandas_dataframeflytekit/types/structured/structured_dataset.py", line 351, in _finder
raise ValueError(f"Failed to find a handler for {df_type}, protocol {protocol}, fmt {format}")
Message:
Failed to find a handler for <class 'pandas.core.frame.DataFrame'>, protocol bq, fmt parquettask code
@task
def convert_bq_table_to_pandas_dataframe(sd: DogeCoinDataset) -> pd.DataFrame:
return sd.open(pd.DataFrame).all()input to the task looks like
{
"sd": {
"uri": "bq://<proj-id>:shahid.temp_table_flyte",
"format": "parquet",
"columns": [
{
"hash": "string"
},
{
"size": "integer"
},
{
"block_number": "integer"
}
]
}
}k
Ketan (kumare3)
02/27/2023, 3:45 PMThis seems like a dependency issue. Cc @Yee do you need to install some bq dependency? Is that not included with bq plugin
k
Kevin Su
02/27/2023, 4:38 PMyes, dependency issue. need to install these package. https://github.com/flyteorg/flytekit/blob/d0b72a8ffd9b24233576e33ff043f8392e86021c/flytekit/types/structured/__init__.py#L42
m
Mohd Shahid Khan Afridi
02/27/2023, 6:24 PMThanks @Ketan (kumare3) @Kevin Su 🙏
Including
google-cloud-bigquerygoogle-cloud-bigquery-storage