https://flyte.org logo
#ask-the-community
Title
# ask-the-community
e

ewam

02/03/2023, 10:37 AM
How would I implement "dynamic" workflows like: • network training is running • I add new training data • the new data gets automatically integrated into the training-process or is this out of scope for pipelines and I have to work with my own events?
s

Samhita Alla

02/06/2023, 4:42 AM
If you want to add training data during the execution run, you can do that via signaling:

https://www.youtube.com/watch?v=njNKBke5sQ0

.
e

ewam

02/06/2023, 9:40 AM
@Samhita Alla There might be a misunderstanding here. What I want to achieve is to "always" listen for new data and reconfigure my dataset during runtime. I think "signaling" (at least in flyte terms) is not what I need since it always requires a pre-execution known static DAG.
So I need to implement this myself
s

Samhita Alla

02/06/2023, 9:50 AM
Yes, you ought to have a flow defined prior to registering your DAG. The flow in itself can be dynamic if the structure can be determined only at run time.