https://flyte.org logo
Join Slack
Powered by
Hello everyone, I have a complex etl process in wh...
# flyte-support
c
Hello everyone, I have a complex etl process in which there are more than 1000 dags with complex dependencies (I store them in the database: parent >>child). I decided to try to transfer this to flyte and I don't quite understand whether it is possible to build such complex dependencies. The first image shows the code in af and the graph. I'm trying to do the same in flyte. I want to create a master dag to manage all dependencies. in fact, I want to get one master workflow that will run all the other workflows (in separate files).
f
This is absolutely possible
is the 
get_workflow_dependencies
a task? that you want to run dynamically?
let me share an example that works
if you want them to be in a database then use 
@dynamic
Option 1: if you want to create a static workflow 
master
and not dynamic at runtime - so that you can version view etc. then, check this example for given 
launchplan
it creates a static workflow https://github.com/flyteorg/flytekit/blob/8aaa102d10e957d188503d6e6a51ea845d15db3c/flytekit/remote/backfill.py#L12-L107 It uses remote.fetch https://github.com/flyteorg/flytekit/blob/8aaa102d10e957d188503d6e6a51ea845d15db3c/flytekit/remote/remote.py#L2592-L2595
option 2: Another option is to statically import all the workflows using pythons dynamic import system
My preference would be 1
c
thank you very much, I did something similar.
f
Do share an example when you can
also you might have to raise the max node limit (default is set of 100 - i dont know why)
c
I'll send you an example when I'm next to the macbook
2 Views
Open in Slack
PreviousNext
Powered by