Stefan Avesand
05/04/2022, 12:32 PMsd = StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat')
sd.open(pd.DataFrame).all()
AttributeError: 'NoneType' object has no attribute 'uri'
Samhita Alla
Stefan Avesand
05/04/2022, 1:29 PMfrom flytekit.types.structured import StructuredDataset
Samhita Alla
Stefan Avesand
05/04/2022, 1:53 PMKevin Su
05/04/2022, 1:59 PM@task
def my_task(sd: StructuredDataset) -> StructuredDataset:
return sd
res = my_task(sd=StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat'))
print(res.open(pd.DataFrame).all())
Stefan Avesand
05/04/2022, 2:05 PMSamhita Alla
Kevin Su
05/04/2022, 2:12 PMStefan Avesand
05/04/2022, 2:27 PMfrom google.cloud import bigquery
# Construct a BigQuery client object.
client = bigquery.Client()
query = """
SELECT *
FROM `sp-one-model.quarterly_forecast_2022F1.premium_revenue_tab_input_vat`
LIMIT 20
"""
query_job = client.query(query) # Make an API request.
print("The query data:")
for row in query_job:
# Row values can be accessed by field name or index.
print(row)
Kevin Su
05/04/2022, 2:56 PMStefan Avesand
05/04/2022, 3:02 PMKevin Su
05/04/2022, 3:14 PMbigquery.readsessions.create
permissionUse the BigQuery Storage API to download query results quickly, but at an increased cost. To use this API, first enable it in the Cloud Console. You must also have the bigquery.readsessions.create permission on the project you are billing queries to.
Stefan Avesand
05/04/2022, 3:34 PMSamhita Alla
Stefan Avesand
05/04/2022, 6:06 PMAlekhya
05/04/2022, 6:58 PMSamhita Alla
I think a code example would be really helpful as wellWe have a code example already: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/structured_dataset.html. Will this suffice or are you talking about having a BigQuery-related example?
On that notion, is casting between StructuredDataset and dataframes supported, e.g. like this?Yes!
Stefan Avesand
05/05/2022, 12:04 PMKevin Su
05/05/2022, 1:55 PMStefan Avesand
05/05/2022, 2:58 PM