Brandon Segal
12/05/2022, 2:00 PMdbt compile
to create a fresh copy of manifest.json
(Where dependencies are stored)
2. Reads the model selectors defined in the YAML file
3. Uses the dbt ls
command to list all of the models associated with each model selector in the YAML file
4. Turns the dbt DAG from manifest.json
into a Graph
object with the networkx
library
5. Uses the methods available on the Graph
object to figure out the correct set of dependencies for each group of models defined in the YAML file
6. Writes the dependencies for each group of models (stored as a list of tuples) to file
7. Create an Airflow DAG for each group of models based on the given dependenciesDan Rammer (hamersaw)
12/05/2022, 2:36 PMBrandon Segal
12/05/2022, 2:49 PMDan Rammer (hamersaw)
12/05/2022, 2:52 PMBrandon Segal
12/05/2022, 11:21 PMEduardo Apolinario (eapolinario)
12/05/2022, 11:22 PMBrandon Segal
12/06/2022, 3:51 PM