https://flyte.org logo
Title
r

Robin Eklund

10/03/2022, 9:10 AM
Hi! We are looking into different alternatives how to run DBT with Flyte and came across this plugin: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt Do anyone know the status of it and is it the recommended approach? The alternative we were thinking about is to run a docker image which execute the dbt command. Or maybe there are any other better way of running DBT?
When installing the dbt plugin with
pip install flytekitplugins-dbt
i get this error:
ERROR: Could not find a version that satisfies the requirement flytekitplugins-dbt (from versions: none)
ERROR: No matching distribution found for flytekitplugins-dbt
Also when searching for
flytekitplugins-dbt
in pypi it doesn't give any result
s

Samhita Alla

10/03/2022, 9:51 AM
@Robin Eklund, the plugin isn’t released yet. It’ll be available in flytekit 1.2.0. cc: @Eduardo Apolinario (eapolinario)
r

Robin Eklund

10/03/2022, 9:56 AM
Alright thanks! what is the ETA of releasing flytekit 1.2.0? We were able to install it like this for now:
pip install git+<https://github.com/flyteorg/flytekit.git#subdirectory=plugins/flytekit-dbt>
Do you recommend against using it? or is it possible to use the code in the master branch?
If you would run different DBT jobs how would you implement that with Flyte? So what is the recommended approach? Also a tip, ask DBT to add Flyte here as well: https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production
s

Samhita Alla

10/03/2022, 10:12 AM
Do you recommend against using it? or is it possible to use the code in the master branch?
It’s okay for now but we recommend you to use the stable version after it’s released.
If you would run different DBT jobs how would you implement that with Flyte? So what is the recommended approach?
@Eduardo Apolinario (eapolinario) should be able to answer this.
Also a tip, ask DBT to add Flyte here as well: https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production
Cool, thanks! Would you be up for adding it, Eduardo?
r

Robin Eklund

10/03/2022, 12:13 PM
Thanks for all your answers @Samhita Alla
k

Ketan (kumare3)

10/03/2022, 2:19 PM
The release is sooon, this week mostly
r

Robin Eklund

10/03/2022, 2:20 PM
great, looking forward to it! 😄
@Eduardo Apolinario (eapolinario) or @Samhita Alla When using the DBTRun plugin it works fine if we are not using the
select
option. When doing that we get this error:
raise ValueError("Only generic univariate typing.List[T] type is supported.")
ValueError: Only generic univariate typing.List[T] type is supported.
Is it something you recognize? this is the code:
from flytekitplugins.dbt.task import DBTRun, DBTRunInput
from flytekit import workflow


dbt_task = DBTRun(name="name_of_the_task")


@workflow
def my_wf() -> None:
    _input = DBTRunInput(
        project_dir=f"{project_path}/dbt/analytics",
        profiles_dir=f"{project_path}/dbt/analytics/docker-context",
        profile="default",
        select=['tag:something']
    )

    dbt_task(input=_input)


if __name__ == "__main__":
    my_wf()
k

Ketan (kumare3)

10/04/2022, 2:42 PM
Cc @Eduardo Apolinario (eapolinario) - I think we are incorrectly using list type somewhere instead of the full type
e

Eduardo Apolinario (eapolinario)

10/04/2022, 3:23 PM
@Robin Eklund, looking.
@Robin Eklund, I opened https://github.com/flyteorg/flyte/issues/2953 to track this. Fix incoming. We'll have another flytekit release once this is merged.
r

Robin Eklund

10/06/2022, 10:19 AM
@Eduardo Apolinario (eapolinario) what is the ETA for the DBT plugin fix? (asking so we can plan/decide what solution we should use for now)
e

Eduardo Apolinario (eapolinario)

10/06/2022, 2:59 PM
@Robin Eklund, I expect to have a PR out today. Will have a release once that's merged.
r

Robin Eklund

10/06/2022, 3:00 PM
nice! 🙌
e

Eduardo Apolinario (eapolinario)

10/07/2022, 7:36 AM
@Robin Eklund, can you give flytekit v1.2.1 a try?
r

Robin Eklund

10/07/2022, 7:37 AM
Nice, yes will try it within the next hour
@Eduardo Apolinario (eapolinario) sorry forgot to respond here - i tested it and now the error is not there anymore, thanks! 😄
@Eduardo Apolinario (eapolinario) we created this issue about logging std output when running a DBT task: https://github.com/flyteorg/flyte/issues/3003
k

Ketan (kumare3)

10/19/2022, 1:22 PM
@Robin Eklund how about a contribution? I would also love if we can add something to flytedecks