NYU's Deletion Tuning Matches Full Fine-Tuning by Deleting 0.05% of Weights
Deletion Tuning matches full fine-tuning by zeroing under 0.05% of weights with no other changes ( paper ). Trained adapter is one bit per weight: a 30 MB mask for a 61 GB, 30B MoE model. Prior-task accuracy stays near base while FFT, LoRA, and random sparse tuning drop up to 5.4 points.
Takeaways
- Deletion Tuning matches full fine-tuning by zeroing under 0.05% of weights with no other changes ( paper ).
- Trained adapter is one bit per weight: a 30 MB mask for a 61 GB, 30B MoE model.
- Prior-task accuracy stays near base while FFT, LoRA, and random sparse tuning drop up to 5.4 points.
- Works across RL with verifiable rewards, supervised fine-tuning, and on-policy distillation.
- Masks only the gate projection of transformer blocks; router untouched in MoE models.
- Code, trained masks, and project page available at agenticlearning.ai/deleting-weights .
Researchers at NYU’s Agentic Learning AI Lab and the University of Calgary report that pretrained language models can acquire task-specific behavior by disabling a small set of existing weights. Deletion Tuning freezes the base checkpoint and learns a binary mask over the gate projections in transformer blocks.
The authors report that zeroing fewer than 0.05% of a model’s weights reaches accuracy comparable to full fine-tuning on the largest models they evaluated. Their experiments cover reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation across dense Llama and Qwen models and a 30B mixture-of-experts model.
Hundreds of variants, one frozen checkpoint
Common post-training methods create additional parameters for every specialized model. Full fine-tuning, or FFT, updates many continuous weight values and typically produces another full checkpoint. LoRA stores smaller low-rank matrices, but each adapter still contains newly learned values and can alter capabilities acquired during pretraining.
| Method | Learned change | Deployment artifact | Main constraint |
|---|---|---|---|
| Full fine-tuning | Continuous updates across tuned parameters | Full checkpoint or dense delta | High storage cost per variant |
| LoRA | Continuous low-rank matrices | Adapter or merged checkpoint | Update rank is capped |
| Random sparse tuning | Continuous updates at preselected positions | Sparse delta | Positions are fixed before training |
| Deletion Tuning | Binary keep-or-delete decisions | Bit mask | Target weights retain their original value or become zero |
Compact, non-destructive updates matter when one base model supports many customers, tasks, or sequential training stages. For the 30B mixture-of-experts example, the authors report a 30 MB mask associated with a 61 GB base checkpoint. Each deployment can share the frozen checkpoint and load its own mask.
How a gate becomes a bit mask
A transformer’s feed-forward network expands each token representation, applies nonlinear processing, and projects it back to the model dimension. Many current architectures use a gated feed-forward network in which a gate projection controls how strongly intermediate features pass through the block.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
