Brandon Segal
7 months agoHi! I’m Brandon Segal an Analytics Engineer over at Spotify looking to migrate some of our workflows from Luigi to Flyte! If anyone has experience using flyte with GCP workloads and big query I’d love to hear more about how you did that!
Ketan (kumare3)
7 months agoHi @Brandon Segal - happy new year and welcome to the community- Brandon any specific questions?
Brandon Segal
7 months agoSo I’ve been able to wrap my head around the base case when authoring tasks and workflows with flyte kit. I’ve had a little bit of an issue understanding how to take advantage of backend plugins installed on flyte propeller that don’t have corresponding tasks in flyte kit. Is that a case where we would use a reference task to take advantage of these. In particular I’m asking in regards to the bigquery plugin since a majority of my workflows would involve checking for some table or expectation in bigqueryKetan (kumare3)
7 months ago@Brandon Segal usually a backend plugin should have a flytekit task. Reference tasks are useful to refer to a previously implemented functionality- As for big query … cc @Guillaume Perchais / @Viktor Gerdin , but @Kevin Su just completed implementation of bigquery python flytekit shim https://github.com/flyteorg/flytekit/pull/789
Brandon Segal
7 months agoAhhh gotcha. So I would use a reference task to incorporate some task that is deployed already to flyte that may have been written by another team. In regards to bigquery, I could write a collection of SQLAlchemy tasks which handle the connection and query the services until this shim is pulled in?Ketan (kumare3)
7 months agoThat should work. Performance of the shim is better though
Join thread in Slack
View count: 1