Question about BQ plugin: (ref doc <https://docs.f...
# announcements
v
Question about BQ plugin: (ref doc https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/gcp/bigquery/index.html) Will the flyte task also include monitoring/checking if results for the query is complete? Most of our queries does take a while to execute, so wondering how will the pods behave in this case?
d
@varsha Parthasarathy great question! So the BQ plugin uses Flytes async web api abstraction in the backend. This means that it doesn't actually start a k8s Pod (because one is not required), rather FlytePropeller submits a BQ job (using a deterministic ID) and then periodically checks if the job has been completed. So long-running queries are certainly supported.
v
wow @Dan Rammer (hamersaw) - this is actually awesome, one more question.
then periodically checks
How frequently is this? Reason: Should we worry about being rate limited by BQ?
d
Yeah, it looks like these are all configurable parameters, but the defaults are 30 second refresh (checking job status) and then 10 QPS with a 100 burst for both read and write operations on the bigquery client.
122 Views