limited-train-92032
05/04/2022, 12:32 PMsd = StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat')
sd.open(pd.DataFrame).all()
AttributeError: 'NoneType' object has no attribute 'uri'
tall-lock-23197
limited-train-92032
05/04/2022, 1:29 PMfrom flytekit.types.structured import StructuredDataset
tall-lock-23197
tall-lock-23197
limited-train-92032
05/04/2022, 1:53 PMglamorous-carpet-83516
05/04/2022, 1:59 PM@task
def my_task(sd: StructuredDataset) -> StructuredDataset:
return sd
res = my_task(sd=StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat'))
print(res.open(pd.DataFrame).all())
limited-train-92032
05/04/2022, 2:05 PMtall-lock-23197
glamorous-carpet-83516
05/04/2022, 2:12 PMlimited-train-92032
05/04/2022, 2:27 PMfrom google.cloud import bigquery
# Construct a BigQuery client object.
client = bigquery.Client()
query = """
SELECT *
FROM `sp-one-model.quarterly_forecast_2022F1.premium_revenue_tab_input_vat`
LIMIT 20
"""
query_job = client.query(query) # Make an API request.
print("The query data:")
for row in query_job:
# Row values can be accessed by field name or index.
print(row)
limited-train-92032
05/04/2022, 2:28 PMglamorous-carpet-83516
05/04/2022, 2:56 PMlimited-train-92032
05/04/2022, 3:02 PMglamorous-carpet-83516
05/04/2022, 3:14 PMbigquery.readsessions.create
permissionglamorous-carpet-83516
05/04/2022, 3:15 PMUse the BigQuery Storage API to download query results quickly, but at an increased cost. To use this API, first enable it in the Cloud Console. You must also have the bigquery.readsessions.create permission on the project you are billing queries to.
limited-train-92032
05/04/2022, 3:34 PMlimited-train-92032
05/04/2022, 3:49 PMlimited-train-92032
05/04/2022, 3:49 PMtall-lock-23197
limited-train-92032
05/04/2022, 6:06 PMearly-rain-275
05/04/2022, 6:58 PMtall-lock-23197
I think a code example would be really helpful as wellWe have a code example already: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/structured_dataset.html. Will this suffice or are you talking about having a BigQuery-related example?
On that notion, is casting between StructuredDataset and dataframes supported, e.g. like this?Yes!
limited-train-92032
05/05/2022, 12:04 PMglamorous-carpet-83516
05/05/2022, 1:55 PMlimited-train-92032
05/05/2022, 2:58 PM