We study machine unlearning in the overparameterized regime, where many different models can interpolate the training data. We propose a new unlearning definition based on the minimum-complexity interpolator over the retained data, and introduce MinNorm-OG, an algorithm enforcing this principle using only model gradients at the original solution. We provide theoretical guarantees for several model classes and demonstrate that MinNorm-OG outperforms existing baselines in practice.

Foundation models learn rich representations, but require further training for adaptation to downstream tasks. We introduce a generic PEFT-based meta-learning framework that prepares models to adapt efficiently to unseen tasks, and show that for linear models with LoRA, standard retraining is provably suboptimal while our method achieves strict performance guarantees. Experiments on synthetic, vision, and language tasks confirm significant gains over conventional retraining.