hello guys, I need to build a task that saves the ...
# flyte-support
a
hello guys, I need to build a task that saves the data output from the processing in a table in BigQuery on Google cloud. can you help me? Do you have an operator ready or something?
f
a
Yeah! There are doc about how to query BigQuery table, but i'd like to create/update a BigQuery table...
f
Hmm cc @better-air-29727 / @average-cpu-32189 Or @most-gold-65483 - I am not a bq expert Also cc @glamorous-carpet-83516
e
I don’t recall the existing plugin supporting that
for the custom-code approach, checkout: https://github.com/googleapis/python-bigquery-pandas
There are these settings https://cloud.google.com/bigquery/docs/writing-results , but I don’t think you’re talking about writing the results of a bigquery query back to bq. [ which also I don’t recall as existing options in the plugin — I haven’t looked at that recently - and if doesn’t exist, could be extended to include it ]
f
i think structured-datasets directly support persistence of pandas to bigquery
I know @famous-businessperson-24711 and team use that
e
That looks like will do what I assume @abundant-night-96152 is seeking
👍 1
m
For us, so far we have been using BQ either using https://flyte-org.slack.com/archives/CP2HDHKE1/p1655157364872009?thread_ts=1655143420.973889&cid=CP2HDHKE1 or via DBT. @abundant-translator-2949 is going to explore the flyte bq plugin in the next several months https://flyte-org.slack.com/archives/CP2HDHKE1/p1655159503300479?thread_ts=1655143420.973889&cid=CP2HDHKE1
❤️ 2
g
Here is an example to write panda dataframe to bq. https://gist.github.com/pingsutw/56d56ce5aea997586cce30ab659859fa
👍 1
a
Amazing, guys! Helped me a lot! Thanks!
f
Could you help improve docs, where you can
1
b
163 Views