Is it possible to create a task that returns a flyte workflow? I was interested to see if I could replicate the logic in this proposed Airlfow DAG orchestration that orchestrates a dbt project. In the workflow they do the following: 1. Runs

dbt compile

to create a fresh copy of

manifest.json

(Where dependencies are stored) 2. Reads the model selectors defined in the YAML file 3. Uses the

dbt ls

command to list all of the models associated with each model selector in the YAML file 4. Turns the dbt DAG from

manifest.json

into a

Graph

object with the

networkx

library 5. Uses the methods available on the

Graph

object to figure out the correct set of dependencies for each group of models defined in the YAML file 6. Writes the dependencies for each group of models (stored as a list of tuples) to file 7. Create an Airflow DAG for each group of models based on the given dependencies

Hey @Brandon Segal, this is a great question! Do you have a link to the example? Also, is the goal here to run the compiled task once or do you want to register it so it's available in the Flyte UI?

Oh of course here is the article here. We would take the flyte workflow output from this script and register it because I think that typically the DAG would only change on code updates. I wonder how it would work since it would need to be able to take a workflow output and then run that or use the script for code generation of the workflow?

cc @Eduardo Apolinario (eapolinario) - interesting dbt use-case. ideas on the best way to integration something similar in Flyte?

Similarly I wonder if it would make sense to update the api for the dbt plugin to be more in line with other integrations like Prefect and Airflow where it is a single task type that wraps a command line argument

This looks very interesting. I can see a similar flow working in Flyte. @Brandon Segal, mind filing an issue to discuss this? We might go this way of leaning on @dynamic to build the graph

Just added a issue here @Eduardo Apolinario (eapolinario).