Thread
#introductions
    Brandon Segal

    Brandon Segal

    8 months ago
    Hi! I’m Brandon Segal an Analytics Engineer over at Spotify looking to migrate some of our workflows from Luigi to Flyte! If anyone has experience using flyte with GCP workloads and big query I’d love to hear more about how you did that!
    Ketan (kumare3)

    Ketan (kumare3)

    8 months ago
    Hi @Brandon Segal - happy new year and welcome to the community
    Brandon any specific questions?
    Brandon Segal

    Brandon Segal

    8 months ago
    So I’ve been able to wrap my head around the base case when authoring tasks and workflows with flyte kit. I’ve had a little bit of an issue understanding how to take advantage of backend plugins installed on flyte propeller that don’t have corresponding tasks in flyte kit. Is that a case where we would use a reference task to take advantage of these. In particular I’m asking in regards to the bigquery plugin since a majority of my workflows would involve checking for some table or expectation in bigquery
    Ketan (kumare3)

    Ketan (kumare3)

    8 months ago
    @Brandon Segal usually a backend plugin should have a flytekit task. Reference tasks are useful to refer to a previously implemented functionality
    As for big query … cc @Guillaume Perchais / @Viktor Gerdin , but @Kevin Su just completed implementation of bigquery python flytekit shim https://github.com/flyteorg/flytekit/pull/789
    Brandon Segal

    Brandon Segal

    8 months ago
    Ahhh gotcha. So I would use a reference task to incorporate some task that is deployed already to flyte that may have been written by another team. In regards to bigquery, I could write a collection of SQLAlchemy tasks which handle the connection and query the services until this shim is pulled in?
    Ketan (kumare3)

    Ketan (kumare3)

    8 months ago
    That should work. Performance of the shim is better though