Hi! I’m Brandon Segal an Analytics Engineer over at Spotify looking to migrate some of our workflows from Luigi to Flyte! If anyone has experience using flyte with GCP workloads and big query I’d love to hear more about how you did that!

Hi @bored-laptop-29637 - happy new year and welcome to the community

So I’ve been able to wrap my head around the base case when authoring tasks and workflows with flyte kit. I’ve had a little bit of an issue understanding how to take advantage of backend plugins installed on flyte propeller that don’t have corresponding tasks in flyte kit. Is that a case where we would use a reference task to take advantage of these. In particular I’m asking in regards to the bigquery plugin since a majority of my workflows would involve checking for some table or expectation in bigquery

@bored-laptop-29637 usually a backend plugin should have a flytekit task. Reference tasks are useful to refer to a previously implemented functionality

Ahhh gotcha. So I would use a reference task to incorporate some task that is deployed already to flyte that may have been written by another team. In regards to bigquery, I could write a collection of SQLAlchemy tasks which handle the connection and query the services until this shim is pulled in?

That should work. Performance of the shim is better though