Question about BQ plugin: (ref doc https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/gcp/bigquery/index.html) Will the flyte task also include monitoring/checking if results for the query is complete? Most of our queries does take a while to execute, so wondering how will the pods behave in this case?

@varsha Parthasarathy great question! So the BQ plugin uses Flytes async web api abstraction in the backend. This means that it doesn't actually start a k8s Pod (because one is not required), rather FlytePropeller submits a BQ job (using a deterministic ID) and then periodically checks if the job has been completed. So long-running queries are certainly supported.

wow @Dan Rammer (hamersaw) - this is actually awesome, one more question.

then periodically checks

How frequently is this? Reason: Should we worry about being rate limited by BQ?

Yeah, it looks like these are all configurable parameters, but the defaults are 30 second refresh (checking job status) and then 10 QPS with a 100 burst for both read and write operations on the bigquery client.