I am running into an issue with the flytesnacks bi...
# ask-the-communitye
Evan Sadler
05/31/2023, 5:24 PMI am running into an issue with the flytesnacks bigquery example that I can't figure out.
Copy code
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 530, in dispatch_execute
raise type(exc)(msg) from exc
Message:
Failed to convert inputs of task 'flytesnacks.convert_bq_table_to_pandas_dataframe':
Protocol not known: bq
SYSTEM ERROR! Contact platform administrators.Evan Sadler
05/31/2023, 5:25 PMhere is the code! It seems like I might need to install an extra package or enable something. I would appreciate some help 🙂
Copy code
DogeCoinDataset = Annotated[
StructuredDataset, kwtypes(hash=str, size=int, block_number=int)
]
bigquery_task_templatized_query = BigQueryTask(
name="sql.bigquery.w_io",
# Define inputs as well as their types that can be used to customize the query.
inputs=kwtypes(version=int),
output_structured_dataset_type=pd.DataFrame,
task_config=BigQueryConfig(ProjectID=""),
query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;",
)
# %%
# StructuredDataset transformer can convert query result to pandas dataframe here.
# We can also change "pandas.dataframe" to "pyarrow.Table", and convert result to Arrow table.
@task(container_image=python_image)
def convert_bq_table_to_pandas_dataframe(sd: pd.DataFrame):
print(sd)j
jeev
05/31/2023, 5:32 PM
what version of flytekit?
e
Evan Sadler
05/31/2023, 5:34 PMThis is what I am doing locally
Copy code
flytekit==1.6.2
flytekitplugins-bigquery==1.6.1j
jeev
05/31/2023, 5:35 PM
jeev
05/31/2023, 5:35 PM
jeev
05/31/2023, 5:42 PM
@Evan Sadler: maybe just try adding
Copy code
import google.cloud.bigquerye
Evan Sadler
05/31/2023, 5:45 PMI am trying now thank you 🙏
Evan Sadler
05/31/2023, 7:02 PMI had to install the bigquery deps for it to work
Copy code
python_image = ImageSpec(name="python",
base_image="python:3.10-slim-buster",
registry="registry",
packages=["flytekitplugins-pandera","flytekitplugins-bigquery" ,"pandas", "numpy", "pretty_html_table", "google-cloud-bigquery-storage", "google-cloud-bigquery"])a
Ariel Kaspit
07/04/2023, 2:21 PM@Nizar Hattab look what I found
152 Views