hello guys, I need to build a task that saves the ...
# ask-the-community
s
hello guys, I need to build a task that saves the data output from the processing in a table in BigQuery on Google cloud. can you help me? Do you have an operator ready or something?
k
s
Yeah! There are doc about how to query BigQuery table, but i'd like to create/update a BigQuery table...
k
Hmm cc @Nelson Arapé / @Viktor Gerdin Or @Pradithya Aria Pura - I am not a bq expert Also cc @Kevin Su
a
I don’t recall the existing plugin supporting that
for the custom-code approach, checkout: https://github.com/googleapis/python-bigquery-pandas
There are these settings https://cloud.google.com/bigquery/docs/writing-results , but I don’t think you’re talking about writing the results of a bigquery query back to bq. [ which also I don’t recall as existing options in the plugin — I haven’t looked at that recently - and if doesn’t exist, could be extended to include it ]
k
i think structured-datasets directly support persistence of pandas to bigquery
I know @Dylan Wilder and team use that
a
That looks like will do what I assume @Sérgio de Melo Barreto Junior is seeking
👍 1
p
For us, so far we have been using BQ either using https://flyte-org.slack.com/archives/CP2HDHKE1/p1655157364872009?thread_ts=1655143420.973889&cid=CP2HDHKE1 or via DBT. @Arief Rahmansyah is going to explore the flyte bq plugin in the next several months https://flyte-org.slack.com/archives/CP2HDHKE1/p1655159503300479?thread_ts=1655143420.973889&cid=CP2HDHKE1
❤️ 2
k
Here is an example to write panda dataframe to bq. https://gist.github.com/pingsutw/56d56ce5aea997586cce30ab659859fa
👍 1
s
Amazing, guys! Helped me a lot! Thanks!
k
Could you help improve docs, where you can
1
s
163 Views