Is it possible to create a task that returns a fly...
# ask-the-community
b
Is it possible to create a task that returns a flyte workflow? I was interested to see if I could replicate the logic in this proposed Airlfow DAG orchestration that orchestrates a dbt project. In the workflow they do the following: 1. Runs
dbt compile
to create a fresh copy of
manifest.json
(Where dependencies are stored) 2. Reads the model selectors defined in the YAML file 3. Uses the
dbt ls
command to list all of the models associated with each model selector in the YAML file 4. Turns the dbt DAG from
manifest.json
into a
Graph
object with the
networkx
library 5. Uses the methods available on the
Graph
object to figure out the correct set of dependencies for each group of models defined in the YAML file 6. Writes the dependencies for each group of models (stored as a list of tuples) to file 7. Create an Airflow DAG for each group of models based on the given dependencies
d
Hey @Brandon Segal, this is a great question! Do you have a link to the example? Also, is the goal here to run the compiled task once or do you want to register it so it's available in the Flyte UI?
I'm wondering if we can use a dynamic workflow for this, because these are already designed to dynamically compile a Flyte DAG and execute as part of the workflow evaluation.
b
Oh of course here is the article here. We would take the flyte workflow output from this script and register it because I think that typically the DAG would only change on code updates. I wonder how it would work since it would need to be able to take a workflow output and then run that or use the script for code generation of the workflow?
d
cc @Eduardo Apolinario (eapolinario) - interesting dbt use-case. ideas on the best way to integration something similar in Flyte?
b
Similarly I wonder if it would make sense to update the api for the dbt plugin to be more in line with other integrations like Prefect and Airflow where it is a single task type that wraps a command line argument
e
This looks very interesting. I can see a similar flow working in Flyte. @Brandon Segal, mind filing an issue to discuss this? We might go this way of leaning on @dynamic to build the graph
b
Just added a issue here @Eduardo Apolinario (eapolinario).