I am trying to run basic BigQueryTask example provided in <h Flyte #ask-the-community

I am trying to run basic BigQueryTask example prov...

Mohd Shahid Khan Afridi

02/27/2023, 1:27 PM

I am trying to run basic BigQueryTask example provided in bq plugin doc, when it runs the

convert_bq_table_to_pandas_dataframe

task below error is thrown. Is this combination not supported in structure_dataset encoding decoding?

Copy code

flytekit/types/structured/structured_dataset.py", line 351, in _finder
        raise ValueError(f"Failed to find a handler for {df_type}, protocol {protocol}, fmt {format}")
Message:
    Failed to find a handler for <class 'pandas.core.frame.DataFrame'>, protocol bq, fmt parquet

[RESOLVED]

Mohd Shahid Khan Afridi

02/27/2023, 1:27 PM

task code

Copy code

@task
def convert_bq_table_to_pandas_dataframe(sd: DogeCoinDataset) -> pd.DataFrame:
    return sd.open(pd.DataFrame).all()

Mohd Shahid Khan Afridi

02/27/2023, 1:31 PM

input to the task looks like

Copy code

{
  "sd": {
    "uri": "bq://<proj-id>:shahid.temp_table_flyte",
    "format": "parquet",
    "columns": [
      {
        "hash": "string"
      },
      {
        "size": "integer"
      },
      {
        "block_number": "integer"
      }
    ]
  }
}

Ketan (kumare3)

02/27/2023, 3:45 PM

This seems like a dependency issue. Cc @Yee do you need to install some bq dependency? Is that not included with bq plugin

Kevin Su

02/27/2023, 4:38 PM

yes, dependency issue. need to install these package. https://github.com/flyteorg/flytekit/blob/d0b72a8ffd9b24233576e33ff043f8392e86021c/flytekit/types/structured/__init__.py#L42

Mohd Shahid Khan Afridi

02/27/2023, 6:24 PM

Thanks @Ketan (kumare3) @Kevin Su 🙏 Including

google-cloud-bigquery

and

google-cloud-bigquery-storage

in requirement file resolved the problem

151 Views

Open in Slack

Previous Next