<#721 Example: add async training with periodic ev...
# flyte-github
a
#721 Example: add async training with periodic eval and early stopping exa… Pull request opened by kumare3 …mple Add examples/ml/async_train_eval.py demonstrating a common ML pattern where a long-running training task runs concurrently with periodic evaluation rounds that can trigger early stopping on convergence. How it works: • A training task runs in the background via asyncio.create_task, writing JSON checkpoints (epoch + loss) to a shared File location after each epoch, and returning the final checkpoint on completion. • The orchestrator loops through evaluation rounds, using flyte.durable.sleep.aio (crash-resilient) between each round. The sleep is raced against the training task via asyncio.wait so that training completion or failure is detected immediately. • Each eval round reads the latest checkpoint and launches a batch inference task that checks if loss is below a convergence threshold. • On convergence: training is cancelled and the pipeline returns early. • On training failure: any in-progress eval is cancelled and the exception propagates. • On training completion: a final eval is run against the checkpoint returned by the training task. All tuning knobs (epochs, epoch duration, convergence threshold, eval interval, max eval rounds) are exposed as parameters with defaults. To run it flyte run examples/ml/async_train_eval.py main flyteorg/flyte-sdk