DistilRepair: Utility-Preserving Unlearning via Distilled Gradient Repair

Abstract

Machine unlearning aims to remove the influence of designated training data from a trained model without incurring the full computational cost of retraining from scratch. This objective is challenging because approximate unlearning methods must balance effective forgetting against preservation of retained utility, and the difficulty becomes greater in sparse neural networks where reduced parameter capacity can amplify instability. This Study addresses these challenges by proposing DistilRepair, an anchored distillation framework for utility-preserving approximate unlearning in both dense and sparse models. DistilRepair combines targeted forget-side optimization with retain-side repair guided by a frozen teacher, while parameter anchoring constrains harmful deviation from the pre-unlearning solution. The method further incorporates staged classifier-focused warmup, Batch Normalization freezing during forget updates, and numerical-stability safeguards to improve robustness, particularly under sparsity. Experiments on the HAM10000 skin lesion dataset with EfficientNet-B0 and the CIFAR-10 dataset with ResNet-18 show that DistilRepair consistently narrows the gap between approximate unlearning and gold-standard retraining. On HAM10000, DistilRepair achieves near-retraining performance for dense class-wise forgetting and strong sparse unlearning performance while reducing runtime by approximately 11 times. On CIFAR-10, the method remains close to retraining under class-wise forgetting and shows strong utility preservation under random forgetting, while preserving competitive retained and test accuracy. Across both datasets and model families, DistilRepair reduces unlearning time by a large margin relative to retraining, demonstrating that anchored distillation is an effective and practical strategy for approximate unlearning in dense and sparse neural networks.