ewam02/06/2023, 8:01 AM
tasks and run them as long es condition is met in
Option B: One "giant" workflow without
@task def train(): ... @task def eval(): ... @dynamic def job(): running = True while running: train_result = train() score = eval() if score > 0.5: running = False
Are there advantages of one vs. the other? Would it be problematic that in Option A the DAG might grow very big?
def train(): ... def eval(): ... @workflow def job(): running = True while running: train_result = train() score = eval() if score > 0.5: running = False
Bernhard Stadlbauer02/06/2023, 8:15 AM
task from your example, I like to think of of them us one node in the workflow graph wich can produce another (fixed) graph at workflow runtime. So while this gives you some dynamism, it is still a workflow that get’s produced once (which is why the
will most likely not work, as this leads to an infinitely sized graph). You might get around this using a fixed sized loop and conditionals
ewam02/06/2023, 8:37 AM
are not defined as tasks!
Bernhard Stadlbauer02/06/2023, 8:39 AM
(as it is just a description of a graph (DAG)). So you could wrap all of this in a
instead of a
, but this might not be what you want
ewam02/06/2023, 8:41 AM
of course. Sorry about the confusion.
Bernhard Stadlbauer02/06/2023, 8:49 AM
ewam02/06/2023, 9:00 AM
would give me the ability for parallel execution (i.e. save time for the customer)
Niels Bantilan02/06/2023, 2:51 PM
ewam02/08/2023, 7:43 AM
, I'm 100% convinced it should be implemented as a workflow. Training a model could need the following tasks: • data-loading • data-processing • model-loading • model-pretraining (could be even a sub-workflow) • architecture-search / auto-ml / hyperparameter-search • training on the training dataset • evaluation on the eval dataset • model conversion / export / storage Just realized that your might just be about
as a task vs workflow: In this case it should be a task I think.
Niels Bantilan02/08/2023, 3:14 PM
ewam02/08/2023, 4:30 PM
do you mean it in the sense the you stop early during a tight training loop of a single model, or in a hyperparameter tuning context?I mean it in the sense of "customer is satisfied with dice-score >= 0.8 -> no need to train further and pay for ressources"
I also think the above workflow is pretty static. Hyperparameter tuning can be implemented as ray above?yes, it is. Forgive me if the question was specially for a "dynamic" workflow. Than I can put up another example (e.g. what we do currently in prod when training networks for customers).
Niels Bantilan02/08/2023, 4:41 PM
I mean it in the sense of “customer is satisfied with dice-score >= 0.8 -> no need to train further and pay for ressources”cool, so just to clarify: the
is something that the customer determines for some hyperparameter configuration?