https://flyte.org logo
Join Slack
Powered by
Hi! We are looking into different alternatives how...
# ask-the-community
r
Hi! We are looking into different alternatives how to run DBT with Flyte and came across this plugin: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt Do anyone know the status of it and is it the recommended approach? The alternative we were thinking about is to run a docker image which execute the dbt command. Or maybe there are any other better way of running DBT?
When installing the dbt plugin with 
pip install flytekitplugins-dbt
i get this error:
Copy code
ERROR: Could not find a version that satisfies the requirement flytekitplugins-dbt (from versions: none)
ERROR: No matching distribution found for flytekitplugins-dbt
Also when searching for 
flytekitplugins-dbt
in pypi it doesn't give any result
s
@Robin Eklund, the plugin isn’t released yet. It’ll be available in flytekit 1.2.0. cc: @Eduardo Apolinario (eapolinario)
r
Alright thanks! what is the ETA of releasing flytekit 1.2.0? We were able to install it like this for now: 
pip install git+<https://github.com/flyteorg/flytekit.git#subdirectory=plugins/flytekit-dbt>
Do you recommend against using it? or is it possible to use the code in the master branch?
If you would run different DBT jobs how would you implement that with Flyte? So what is the recommended approach? Also a tip, ask DBT to add Flyte here as well: https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production
s
Do you recommend against using it? or is it possible to use the code in the master branch?
It’s okay for now but we recommend you to use the stable version after it’s released.
If you would run different DBT jobs how would you implement that with Flyte? So what is the recommended approach?
@Eduardo Apolinario (eapolinario) should be able to answer this.
Also a tip, ask DBT to add Flyte here as well: https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production
Cool, thanks! Would you be up for adding it, Eduardo?
r
Thanks for all your answers @Samhita Alla
k
The release is sooon, this week mostly
r
great, looking forward to it! 😄
@Eduardo Apolinario (eapolinario) or @Samhita Alla When using the DBTRun plugin it works fine if we are not using the 
select
option. When doing that we get this error:
Copy code
raise ValueError("Only generic univariate typing.List[T] type is supported.")
ValueError: Only generic univariate typing.List[T] type is supported.
Is it something you recognize? this is the code:
Copy code
from flytekitplugins.dbt.task import DBTRun, DBTRunInput
from flytekit import workflow


dbt_task = DBTRun(name="name_of_the_task")


@workflow
def my_wf() -> None:
    _input = DBTRunInput(
        project_dir=f"{project_path}/dbt/analytics",
        profiles_dir=f"{project_path}/dbt/analytics/docker-context",
        profile="default",
        select=['tag:something']
    )

    dbt_task(input=_input)


if __name__ == "__main__":
    my_wf()
k
Cc @Eduardo Apolinario (eapolinario) - I think we are incorrectly using list type somewhere instead of the full type
e
@Robin Eklund, looking.
@Robin Eklund, I opened https://github.com/flyteorg/flyte/issues/2953 to track this. Fix incoming. We'll have another flytekit release once this is merged.
r
@Eduardo Apolinario (eapolinario) what is the ETA for the DBT plugin fix? (asking so we can plan/decide what solution we should use for now)
e
@Robin Eklund, I expect to have a PR out today. Will have a release once that's merged.
r
nice! 🙌
e
@Robin Eklund, can you give flytekit v1.2.1 a try?
r
Nice, yes will try it within the next hour
@Eduardo Apolinario (eapolinario) sorry forgot to respond here - i tested it and now the error is not there anymore, thanks! 😄
@Eduardo Apolinario (eapolinario) we created this issue about logging std output when running a DBT task: https://github.com/flyteorg/flyte/issues/3003
k
@Robin Eklund how about a contribution? I would also love if we can add something to flytedecks
159 Views
Open in Slack
PreviousNext
Powered by