How would I implement "dynamic" workflows like:
• network training is running
• I add new training data
• the new data gets automatically integrated into the training-process
or is this out of scope for pipelines and I have to work with my own events?
02/06/2023, 4:42 AM
If you want to add training data during the execution run, you can do that via signaling:
02/06/2023, 9:40 AM
@Samhita Alla There might be a misunderstanding here. What I want to achieve is to "always" listen for new data and reconfigure my dataset during runtime.
I think "signaling" (at least in flyte terms) is not what I need since it always requires a pre-execution known static DAG.
So I need to implement this myself
02/06/2023, 9:50 AM
Yes, you ought to have a flow defined prior to registering your DAG. The flow in itself can be dynamic if the structure can be determined only at run time.