How would I implement "dynamic" workflows like: - ...
# ask-the-community
e
How would I implement "dynamic" workflows like: • network training is running • I add new training data • the new data gets automatically integrated into the training-process or is this out of scope for pipelines and I have to work with my own events?
s
If you want to add training data during the execution run, you can do that via signaling:

https://www.youtube.com/watch?v=njNKBke5sQ0

.
e
@Samhita Alla There might be a misunderstanding here. What I want to achieve is to "always" listen for new data and reconfigure my dataset during runtime. I think "signaling" (at least in flyte terms) is not what I need since it always requires a pre-execution known static DAG.
So I need to implement this myself
s
Yes, you ought to have a flow defined prior to registering your DAG. The flow in itself can be dynamic if the structure can be determined only at run time.
151 Views