https://flyte.org logo
s

Sérgio de Melo Barreto Junior

06/13/2022, 6:03 PM
hello guys, I need to build a task that saves the data output from the processing in a table in BigQuery on Google cloud. can you help me? Do you have an operator ready or something?
k

Ketan (kumare3)

06/13/2022, 7:33 PM
s

Sérgio de Melo Barreto Junior

06/13/2022, 8:02 PM
Yeah! There are doc about how to query BigQuery table, but i'd like to create/update a BigQuery table...
k

Ketan (kumare3)

06/13/2022, 9:35 PM
Hmm cc @Nelson Arapé / @Viktor Gerdin Or @Pradithya Aria Pura - I am not a bq expert Also cc @Kevin Su
a

austin

06/13/2022, 9:54 PM
I don’t recall the existing plugin supporting that
for the custom-code approach, checkout: https://github.com/googleapis/python-bigquery-pandas
There are these settings https://cloud.google.com/bigquery/docs/writing-results , but I don’t think you’re talking about writing the results of a bigquery query back to bq. [ which also I don’t recall as existing options in the plugin — I haven’t looked at that recently - and if doesn’t exist, could be extended to include it ]
k

Ketan (kumare3)

06/13/2022, 10:25 PM
i think structured-datasets directly support persistence of pandas to bigquery
I know @Dylan Wilder and team use that
a

austin

06/13/2022, 10:39 PM
That looks like will do what I assume @Sérgio de Melo Barreto Junior is seeking
👍 1
p

Pradithya Aria Pura

06/14/2022, 1:24 AM
For us, so far we have been using BQ either using https://flyte-org.slack.com/archives/CP2HDHKE1/p1655157364872009?thread_ts=1655143420.973889&cid=CP2HDHKE1 or via DBT. @Arief Rahmansyah is going to explore the flyte bq plugin in the next several months https://flyte-org.slack.com/archives/CP2HDHKE1/p1655159503300479?thread_ts=1655143420.973889&cid=CP2HDHKE1
❤️ 2
k

Kevin Su

06/14/2022, 4:53 AM
Here is an example to write panda dataframe to bq. https://gist.github.com/pingsutw/56d56ce5aea997586cce30ab659859fa
👍 1
s

Sérgio de Melo Barreto Junior

06/14/2022, 2:21 PM
Amazing, guys! Helped me a lot! Thanks!
k

Ketan (kumare3)

06/14/2022, 2:33 PM
Could you help improve docs, where you can
1
s

Smriti Satyan

06/15/2022, 7:18 AM
2 Views