https://flyte.org
Join Slack
Hi! We are looking into different alternatives how to run DBT with Flyte and came across this plugin...
# flyte-support
s
Hi! We are looking into different alternatives how to run DBT with Flyte and came across this plugin: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt Do anyone know the status of it and is it the recommended approach? The alternative we were thinking about is to run a docker image which execute the dbt command. Or maybe there are any other better way of running DBT?
When installing the dbt plugin with 
pip install flytekitplugins-dbt
i get this error:
Copy code
ERROR: Could not find a version that satisfies the requirement flytekitplugins-dbt (from versions: none)
ERROR: No matching distribution found for flytekitplugins-dbt
Also when searching for 
flytekitplugins-dbt
in pypi it doesn't give any result
t
@silly-jelly-43096, the plugin isn’t released yet. It’ll be available in flytekit 1.2.0. cc: @high-accountant-32689
s
Alright thanks! what is the ETA of releasing flytekit 1.2.0? We were able to install it like this for now: 
pip install git+<https://github.com/flyteorg/flytekit.git#subdirectory=plugins/flytekit-dbt>
Do you recommend against using it? or is it possible to use the code in the master branch?
If you would run different DBT jobs how would you implement that with Flyte? So what is the recommended approach? Also a tip, ask DBT to add Flyte here as well: https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production
t
Do you recommend against using it? or is it possible to use the code in the master branch?
It’s okay for now but we recommend you to use the stable version after it’s released.
If you would run different DBT jobs how would you implement that with Flyte? So what is the recommended approach?
@high-accountant-32689 should be able to answer this.
Also a tip, ask DBT to add Flyte here as well: https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production
Cool, thanks! Would you be up for adding it, Eduardo?
👍 1
s
Thanks for all your answers @tall-lock-23197
f
The release is sooon, this week mostly
👍 1
s
great, looking forward to it! 😄
@high-accountant-32689 or @tall-lock-23197 When using the DBTRun plugin it works fine if we are not using the 
select
option. When doing that we get this error:
Copy code
raise ValueError("Only generic univariate typing.List[T] type is supported.")
ValueError: Only generic univariate typing.List[T] type is supported.
Is it something you recognize? this is the code:
Copy code
from flytekitplugins.dbt.task import DBTRun, DBTRunInput
from flytekit import workflow


dbt_task = DBTRun(name="name_of_the_task")


@workflow
def my_wf() -> None:
    _input = DBTRunInput(
        project_dir=f"{project_path}/dbt/analytics",
        profiles_dir=f"{project_path}/dbt/analytics/docker-context",
        profile="default",
        select=['tag:something']
    )

    dbt_task(input=_input)


if __name__ == "__main__":
    my_wf()
f
Cc @high-accountant-32689 - I think we are incorrectly using list type somewhere instead of the full type
h
@silly-jelly-43096, looking.
👍 1
@silly-jelly-43096, I opened https://github.com/flyteorg/flyte/issues/2953 to track this. Fix incoming. We'll have another flytekit release once this is merged.
👍 1
s
@high-accountant-32689 what is the ETA for the DBT plugin fix? (asking so we can plan/decide what solution we should use for now)
h
@silly-jelly-43096, I expect to have a PR out today. Will have a release once that's merged.
🙏 1
s
nice! 🙌
h
@silly-jelly-43096, can you give flytekit v1.2.1 a try?
👍 1
s
Nice, yes will try it within the next hour
@high-accountant-32689 sorry forgot to respond here - i tested it and now the error is not there anymore, thanks! 😄
🎉 1
@high-accountant-32689 we created this issue about logging std output when running a DBT task: https://github.com/flyteorg/flyte/issues/3003
f
@silly-jelly-43096 how about a contribution? I would also love if we can add something to flytedecks
158 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by