Good morning team! Trying to use the new StructuredDataset type to read data from BQ but can’t figure it out.

sd = StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat')
sd.open(pd.DataFrame).all()

AttributeError: 'NoneType' object has no attribute 'uri'

Hi @limited-train-92032! What’s the flytekit version you’re using and can you share the import line of yours?

Hi Samhita! I’m using 1.0.0.

from flytekit.types.structured import StructuredDataset

Can you send StructuredDataset to the task as an argument?

Like this?

something like:

@task
def my_task(sd: StructuredDataset) -> StructuredDataset:
    return sd


res = my_task(sd=StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat'))
print(res.open(pd.DataFrame).all())

That seems to work! I get a permission error due to the wrong service account being used. The GOOGLE_APPLICATION_CREDENTIALS environment variable does not seem to get picked up.

Stefan, you can also open and read the dataframe within the task, right @glamorous-carpet-83516?

@limited-train-92032 nice, Could you run some basic example to make sure your google credentials has enough permission to read/write BQ table. Like this one https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-python We use google cloud python SDK, and it should automatically pick up GOOGLE_APPLICATION_CREDENTIALS. @tall-lock-23197 Yeah, we are able to do that.

That example works fine:

from google.cloud import bigquery

# Construct a BigQuery client object.
client = bigquery.Client()

query = """
    SELECT *
    FROM `sp-one-model.quarterly_forecast_2022F1.premium_revenue_tab_input_vat`
    LIMIT 20
"""
query_job = client.query(query)  # Make an API request.

print("The query data:")
for row in query_job:
    # Row values can be accessed by field name or index.
    print(row)

sorry, let me debug it.

Ah, it’s a different service altogether, called the BigQuery Storage Read API

Seems like you IAM role doesn’t have

bigquery.readsessions.create

permission

Thanks, it’s enabled already though

@billowy-sundown-31926/@early-rain-275, can you please add the above to the StructuredDataset API doc?

I think a code example would be really helpful as well 🙂 On that notion, is casting between StructuredDataset and dataframes supported, e.g. like this?

Sure @tall-lock-23197. Will add it in the StructuredDaraset API doc.

I think a code example would be really helpful as well

We have a code example already: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/structured_dataset.html. Will this suffice or are you talking about having a BigQuery-related example?

On that notion, is casting between StructuredDataset and dataframes supported, e.g. like this?

Yes!

A code example showing how to create and use a StructuredDataset from a bq, gs or s3 url was my thought.

Here is an example for bq, https://flyte--747.org.readthedocs.build/projects/cookbook/en/747/auto/integrations/gcp/bigquery/bigquery.html#sphx-glr-auto-int[…]query-bigquery-py

Yes, I looked at that example, but it only shows how to ingest data from BQ using BigQueryTask, not StructuredDataset