Hey folks, Does anyone tried using dbt in flyte workflow ? I am trying a use case to run flyte dbt workflow with dbt plugins? Is there a way to cache the pods directories when you run dbt deps in one step and therefore I no need to run in downstream pods for the build, test and freshness tasks (pods) ?
Also checking this as an idea 💡 https://docs.getdbt.com/reference/programmatic-invocations ? Is flyte plugins is developed on top of this ?
11/10/2023, 2:56 PM
It is not built on this please feel free to propose a change
Sree Rama Chinthamsetty
11/10/2023, 4:10 PM
Thank you Ketan!
How can I cache the result from previous pod (task) to the next task ?
Because, in this case its a target directory having compiled packages installed, compiled SQL and artefacts of dbt.