https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
e

Evan Sadler

05/31/2023, 5:24 PM
I am running into an issue with the flytesnacks bigquery example that I can't figure out.
Copy code
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 530, in dispatch_execute
        raise type(exc)(msg) from exc

Message:

    Failed to convert inputs of task 'flytesnacks.convert_bq_table_to_pandas_dataframe':
  Protocol not known: bq

SYSTEM ERROR! Contact platform administrators.
here is the code! It seems like I might need to install an extra package or enable something. I would appreciate some help 🙂
Copy code
DogeCoinDataset = Annotated[
    StructuredDataset, kwtypes(hash=str, size=int, block_number=int)
]

bigquery_task_templatized_query = BigQueryTask(
    name="sql.bigquery.w_io",
    # Define inputs as well as their types that can be used to customize the query.
    inputs=kwtypes(version=int),
    output_structured_dataset_type=pd.DataFrame,
    task_config=BigQueryConfig(ProjectID=""),
    query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;",
)

# %%
# StructuredDataset transformer can convert query result to pandas dataframe here.
# We can also change "pandas.dataframe" to "pyarrow.Table", and convert result to Arrow table.
@task(container_image=python_image)
def convert_bq_table_to_pandas_dataframe(sd: pd.DataFrame):
    print(sd)
j

jeev

05/31/2023, 5:32 PM
what version of flytekit?
e

Evan Sadler

05/31/2023, 5:34 PM
This is what I am doing locally
Copy code
flytekit==1.6.2
flytekitplugins-bigquery==1.6.1
j

jeev

05/31/2023, 5:35 PM
@Evan Sadler: maybe just try adding
Copy code
import google.cloud.bigquery
to the top of the module?
e

Evan Sadler

05/31/2023, 5:45 PM
I am trying now thank you 🙏
I had to install the bigquery deps for it to work
Copy code
python_image = ImageSpec(name="python",
                        base_image="python:3.10-slim-buster",
                        registry="registry",
                        packages=["flytekitplugins-pandera","flytekitplugins-bigquery" ,"pandas", "numpy", "pretty_html_table", "google-cloud-bigquery-storage", "google-cloud-bigquery"])
a

Ariel Kaspit

07/04/2023, 2:21 PM
@Nizar Hattab look what I found
2 Views
Powered by