#885 implement intratask checkpointing in the sdk Pull request opened by cosmicBboy This pull request introduces a comprehensive suite of example scripts demonstrating how to implement checkpointing and failure recovery for machine learning training workflows using Flyte's

AsyncCheckpoint

mechanism. The examples cover a range of ML frameworks including scikit-learn, PyTorch, PyTorch Lightning, and Hugging Face Transformers, each showing how to persist and restore training state across task retries. The scripts also illustrate best practices for simulating failures and integrating checkpointing into iterative training processes. The most important changes are: Scikit-learn checkpointing: • Added

sklearn_partial_checkpoint.py

to demonstrate incremental training with

SGDClassifier.partial_fit

, using checkpointing to persist and restore model state and training progress across retries. PyTorch checkpointing: • Added

pytorch_task_checkpoint.py

to show checkpointing of a simple PyTorch model's state dict and optimizer at each epoch, with failure simulation and recovery. PyTorch Lightning checkpointing: • Added

pytorch_lightning_checkpoint.py

to illustrate checkpointing for a LightningModule using

ModelCheckpoint

and custom callbacks, including logic for resuming from the latest checkpoint and injecting failures. Hugging Face Transformers checkpointing: • Added

huggingface_trainer_checkpoint.py

to provide an example of checkpointing with

transformers.Trainer

, integrating Flyte checkpointing with Hugging Face's checkpointing and recovery system, and simulating failures at defined intervals. Generic byte checkpointing: • Added

generic_data_checkpoint.py

as a minimal example of checkpointing arbitrary byte data, demonstrating the basic checkpoint load/save/retry pattern. flyteorg/flyte-sdk