https://flyte.org logo
#announcements
Title
# announcements
v

varsha Parthasarathy

09/08/2022, 7:56 PM
Question about BQ plugin: (ref doc https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/gcp/bigquery/index.html) Will the flyte task also include monitoring/checking if results for the query is complete? Most of our queries does take a while to execute, so wondering how will the pods behave in this case?
d

Dan Rammer (hamersaw)

09/08/2022, 8:45 PM
@varsha Parthasarathy great question! So the BQ plugin uses Flytes async web api abstraction in the backend. This means that it doesn't actually start a k8s Pod (because one is not required), rather FlytePropeller submits a BQ job (using a deterministic ID) and then periodically checks if the job has been completed. So long-running queries are certainly supported.
v

varsha Parthasarathy

09/09/2022, 12:09 AM
wow @Dan Rammer (hamersaw) - this is actually awesome, one more question.
then periodically checks
How frequently is this? Reason: Should we worry about being rate limited by BQ?
d

Dan Rammer (hamersaw)

09/09/2022, 9:38 AM
Yeah, it looks like these are all configurable parameters, but the defaults are 30 second refresh (checking job status) and then 10 QPS with a 100 burst for both read and write operations on the bigquery client.