flaky-car-61813
02/06/2023, 8:01 AMtrain
and eval
tasks and run them as long es condition is met in @dynamic
@task
def train():
...
@task
def eval():
...
@dynamic
def job():
running = True
while running:
train_result = train()
score = eval()
if score > 0.5:
running = False
Option B: One "giant" workflow without train
and eval
tasks:
def train():
...
def eval():
...
@workflow
def job():
running = True
while running:
train_result = train()
score = eval()
if score > 0.5:
running = False
Are there advantages of one vs. the other? Would it be problematic that in Option A the DAG might grow very big?agreeable-kitchen-44189
02/06/2023, 8:15 AMdynamic
task from your example, I like to think of of them us one node in the workflow graph wich can produce another (fixed) graph at workflow runtime.
So while this gives you some dynamism, it is still a workflow that get’s produced once (which is why the while True
will most likely not work, as this leads to an infinitely sized graph).
You might get around this using a fixed sized loop and conditionalsflaky-car-61813
02/06/2023, 8:37 AMtrain
and eval
are not defined as tasks!agreeable-kitchen-44189
02/06/2023, 8:39 AMworkflow
(as it is just a description of a graph (DAG)). So you could wrap all of this in a task
instead of a workflow
, but this might not be what you wantflaky-car-61813
02/06/2023, 8:41 AM@task
of course. Sorry about the confusion.agreeable-kitchen-44189
02/06/2023, 8:49 AMflaky-car-61813
02/06/2023, 9:00 AMflaky-car-61813
02/06/2023, 9:00 AMflaky-car-61813
02/06/2023, 9:02 AMtraining
and evalutation
would give me the ability for parallel execution (i.e. save time for the customer)broad-monitor-993
02/06/2023, 2:51 PMbroad-monitor-993
02/06/2023, 2:53 PMflaky-car-61813
02/08/2023, 7:43 AMdef train()
, I'm 100% convinced it should be implemented as a workflow. Training a model could need the following tasks:
• data-loading
• data-processing
• model-loading
• model-pretraining (could be even a sub-workflow)
• architecture-search / auto-ml / hyperparameter-search
• training on the training dataset
• evaluation on the eval dataset
• model conversion / export / storage
Just realized that your might just be about def train()
as a task vs workflow: In this case it should be a task I think.broad-monitor-993
02/08/2023, 3:14 PMfreezing-airport-6809
flaky-car-61813
02/08/2023, 4:30 PMdo you mean it in the sense the you stop early during a tight training loop of a single model, or in a hyperparameter tuning context?I mean it in the sense of "customer is satisfied with dice-score >= 0.8 -> no need to train further and pay for ressources"
flaky-car-61813
02/08/2023, 4:33 PMI also think the above workflow is pretty static. Hyperparameter tuning can be implemented as ray above?yes, it is. Forgive me if the question was specially for a "dynamic" workflow. Than I can put up another example (e.g. what we do currently in prod when training networks for customers).
broad-monitor-993
02/08/2023, 4:41 PMI mean it in the sense of “customer is satisfied with dice-score >= 0.8 -> no need to train further and pay for ressources”cool, so just to clarify: the
dice-score
is something that the customer determines for some hyperparameter configuration?freezing-airport-6809