hello guys I need to build a task that saves the data output

Question

hello guys I need to build a task that saves the data output from the processing in a table in BigQuery on Google cloud can you help me Do you have an operator ready or something

Ketan (kumare3) · Answer

have you looked at <https docs flyte org en latest deployment plugin setup gcp bigquery html>

Sérgio de Melo Barreto Junior · Answer

Yeah There are doc about how to query BigQuery table but i d like to create update a BigQuery table

Ketan (kumare3) · Answer

Hmm cc < Nelson Arapé> < Viktor Gerdin> Or < Pradithya Aria Pura> I am not a bq expert Also cc < Kevin Su>

austin · Answer

I don t recall the existing plugin supporting that

austin · Answer

for the custom code approach checkout <https github com googleapis python bigquery pandas>

austin · Answer

There are these settings <https cloud google com bigquery docs writing results> but I don t think you re talking about writing the results of a bigquery query back to bq which also I don t recall as existing options in the plugin I haven t looked at that recently and if doesn t exist could be extended to include it

Ketan (kumare3) · Answer

i think structured datasets directly support persistence of pandas to bigquery

Ketan (kumare3) · Answer

I know < Dylan Wilder> and team use that

Ketan (kumare3) · Answer

austin · Answer

That looks like will do what I assume < Sérgio de Melo Barreto Junior> is seeking

Pradithya Aria Pura · Answer

For us so far we have been using BQ either using or via DBT < Arief Rahmansyah> is going to explore the flyte bq plugin in the next several months

Kevin Su · Answer

Here is an example to write panda dataframe to bq <https gist github com pingsutw 56d56ce5aea997586cce30ab659859fa>

Sérgio de Melo Barreto Junior · Answer

Amazing guys Helped me a lot Thanks

Ketan (kumare3) · Answer

Could you help improve docs where you can

Smriti Satyan · Answer

We have pushed a PR with the code example < Ketan kumare3 >