I am trying to run basic BigQueryTask example provided in <h Flyte #flyte-support

I am trying to run basic BigQueryTask example prov...

gifted-train-81198

02/27/2023, 1:27 PM

I am trying to run basic BigQueryTask example provided in bq plugin doc, when it runs the

convert_bq_table_to_pandas_dataframe

task below error is thrown. Is this combination not supported in structure_dataset encoding decoding?

Copy code

flytekit/types/structured/structured_dataset.py", line 351, in _finder
        raise ValueError(f"Failed to find a handler for {df_type}, protocol {protocol}, fmt {format}")
Message:
    Failed to find a handler for <class 'pandas.core.frame.DataFrame'>, protocol bq, fmt parquet

[RESOLVED]

gifted-train-81198

02/27/2023, 1:27 PM

task code

Copy code

@task
def convert_bq_table_to_pandas_dataframe(sd: DogeCoinDataset) -> pd.DataFrame:
    return sd.open(pd.DataFrame).all()

gifted-train-81198

02/27/2023, 1:31 PM

input to the task looks like

Copy code

{
  "sd": {
    "uri": "bq://<proj-id>:shahid.temp_table_flyte",
    "format": "parquet",
    "columns": [
      {
        "hash": "string"
      },
      {
        "size": "integer"
      },
      {
        "block_number": "integer"
      }
    ]
  }
}

freezing-airport-6809

02/27/2023, 3:45 PM

This seems like a dependency issue. Cc @thankful-minister-83577 do you need to install some bq dependency? Is that not included with bq plugin

glamorous-carpet-83516

02/27/2023, 4:38 PM

yes, dependency issue. need to install these package. https://github.com/flyteorg/flytekit/blob/d0b72a8ffd9b24233576e33ff043f8392e86021c/flytekit/types/structured/__init__.py#L42

gifted-train-81198

02/27/2023, 6:24 PM

Thanks @freezing-airport-6809 @glamorous-carpet-83516 🙏 Including

google-cloud-bigquery

and

google-cloud-bigquery-storage

in requirement file resolved the problem

👍 1

👍🏽 1

152 Views

Open in Slack

Previous Next