I am running into an issue with the flytesnacks bigquery example that I can't figure out. ```[3/3] c...

billowy-winter-86593

05/31/2023, 5:24 PM

I am running into an issue with the flytesnacks bigquery example that I can't figure out.

Copy code

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 530, in dispatch_execute
        raise type(exc)(msg) from exc

Message:

    Failed to convert inputs of task 'flytesnacks.convert_bq_table_to_pandas_dataframe':
  Protocol not known: bq

SYSTEM ERROR! Contact platform administrators.

billowy-winter-86593

05/31/2023, 5:25 PM

here is the code! It seems like I might need to install an extra package or enable something. I would appreciate some help 🙂

Copy code

DogeCoinDataset = Annotated[
    StructuredDataset, kwtypes(hash=str, size=int, block_number=int)
]

bigquery_task_templatized_query = BigQueryTask(
    name="sql.bigquery.w_io",
    # Define inputs as well as their types that can be used to customize the query.
    inputs=kwtypes(version=int),
    output_structured_dataset_type=pd.DataFrame,
    task_config=BigQueryConfig(ProjectID=""),
    query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;",
)

# %%
# StructuredDataset transformer can convert query result to pandas dataframe here.
# We can also change "pandas.dataframe" to "pyarrow.Table", and convert result to Arrow table.
@task(container_image=python_image)
def convert_bq_table_to_pandas_dataframe(sd: pd.DataFrame):
    print(sd)

freezing-boots-56761

05/31/2023, 5:32 PM

what version of flytekit?

billowy-winter-86593

05/31/2023, 5:34 PM

This is what I am doing locally

Copy code

flytekit==1.6.2
flytekitplugins-bigquery==1.6.1

freezing-boots-56761

05/31/2023, 5:35 PM

suspect an issue here: https://github.com/flyteorg/flytekit/blob/f671fb68092979b0328b97453d0ba0d396a6398e/flytekit/types/structured/bigquery.py#L76

freezing-boots-56761

05/31/2023, 5:35 PM

note the protocol here: https://github.com/flyteorg/flytekit/blob/f671fb68092979b0328b97453d0ba0d396a6398e/flytekit/types/structured/bigquery.py#L19

freezing-boots-56761

05/31/2023, 5:42 PM

@billowy-winter-86593: maybe just try adding

Copy code

import google.cloud.bigquery

to the top of the module?

freezing-boots-56761

05/31/2023, 5:43 PM

https://github.com/flyteorg/flytekit/blob/f671fb68092979b0328b97453d0ba0d396a6398e/flytekit/core/type_engine.py#LL812C2-L812C2

billowy-winter-86593

05/31/2023, 5:45 PM

I am trying now thank you 🙏

billowy-winter-86593

05/31/2023, 7:02 PM

I had to install the bigquery deps for it to work

Copy code

python_image = ImageSpec(name="python",
                        base_image="python:3.10-slim-buster",
                        registry="registry",
                        packages=["flytekitplugins-pandera","flytekitplugins-bigquery" ,"pandas", "numpy", "pretty_html_table", "google-cloud-bigquery-storage", "google-cloud-bigquery"])

elegant-toddler-67101

07/04/2023, 2:21 PM

@important-hamburger-34837 look what I found

152 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.