I am running into an issue with the flytesnacks bi...
# flyte-supportb
billowy-winter-86593
05/31/2023, 5:24 PMI am running into an issue with the flytesnacks bigquery example that I can't figure out.
Copy code
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 530, in dispatch_execute
raise type(exc)(msg) from exc
Message:
Failed to convert inputs of task 'flytesnacks.convert_bq_table_to_pandas_dataframe':
Protocol not known: bq
SYSTEM ERROR! Contact platform administrators.billowy-winter-86593
05/31/2023, 5:25 PMhere is the code! It seems like I might need to install an extra package or enable something. I would appreciate some help 🙂
Copy code
DogeCoinDataset = Annotated[
StructuredDataset, kwtypes(hash=str, size=int, block_number=int)
]
bigquery_task_templatized_query = BigQueryTask(
name="sql.bigquery.w_io",
# Define inputs as well as their types that can be used to customize the query.
inputs=kwtypes(version=int),
output_structured_dataset_type=pd.DataFrame,
task_config=BigQueryConfig(ProjectID=""),
query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;",
)
# %%
# StructuredDataset transformer can convert query result to pandas dataframe here.
# We can also change "pandas.dataframe" to "pyarrow.Table", and convert result to Arrow table.
@task(container_image=python_image)
def convert_bq_table_to_pandas_dataframe(sd: pd.DataFrame):
print(sd)f
freezing-boots-56761
05/31/2023, 5:32 PM
what version of flytekit?
b
billowy-winter-86593
05/31/2023, 5:34 PMThis is what I am doing locally
Copy code
flytekit==1.6.2
flytekitplugins-bigquery==1.6.1f
freezing-boots-56761
05/31/2023, 5:35 PM
freezing-boots-56761
05/31/2023, 5:35 PM
freezing-boots-56761
05/31/2023, 5:42 PM
@billowy-winter-86593: maybe just try adding
Copy code
import google.cloud.bigqueryfreezing-boots-56761
05/31/2023, 5:43 PM
b
billowy-winter-86593
05/31/2023, 5:45 PMI am trying now thank you 🙏
billowy-winter-86593
05/31/2023, 7:02 PMI had to install the bigquery deps for it to work
Copy code
python_image = ImageSpec(name="python",
base_image="python:3.10-slim-buster",
registry="registry",
packages=["flytekitplugins-pandera","flytekitplugins-bigquery" ,"pandas", "numpy", "pretty_html_table", "google-cloud-bigquery-storage", "google-cloud-bigquery"])e
elegant-toddler-67101
07/04/2023, 2:21 PM@important-hamburger-34837 look what I found
153 Views