Soft-MTP Fixes the Noisy Gradient Slowing Down AI Draft Token Training
Soft-MTP identifies and removes a noisy residual term buried in the standard multi-token prediction gradient. The fix uses a KL gradient-recovery trick, implemented as a one-line change to the loss. No new parameters, no measurable overhead, compatible with fused cross-entropy kernels.
Takeaways
- Soft-MTP identifies and removes a noisy residual term buried in the standard multi-token prediction gradient.
- The fix uses a KL gradient-recovery trick, implemented as a one-line change to the loss.
- No new parameters, no measurable overhead, compatible with fused cross-entropy kernels.
- Improves NTP perplexity, zero-shot accuracy, and speculative decoding acceptance from 167M to 1.2B.
- Stronger math and code reasoning after fine-tuning at 1.2B parameters.
- Code and twelve matched checkpoints released at JaydenTeoh/Soft-MTP .
Multi-token prediction (MTP) has become common in large-model pre-training. DeepSeek-V3 and MiniMax-M2 use variants of the technique, which trains auxiliary branches to predict tokens at several future offsets. Those targets increase supervision per sequence and provide draft tokens for speculative decoding.
During speculative decoding, MTP branches propose several tokens and the main model verifies them together. Accepted drafts reduce the number of decoding passes. The new Soft-MTP paper argues that the standard training loss adds a low-quality gradient to this process and presents a compact way to remove it.
Hard labels leak noise into the trunk
Ordinary next-token prediction trains the backbone distribution against the next ground-truth token. Standard MTP applies the same hard-label cross-entropy loss to each auxiliary head, using the ground-truth token at that head’s future offset.
The paper decomposes each MTP cross-entropy gradient into two exact components. One is a KL-divergence term that distills the backbone’s next-token distribution into the corresponding MTP head. This term teaches the auxiliary head to imitate the distribution that will later verify its speculative drafts.
The second component is a residual tied to the hard target. According to the paper’s measurements, its signal-to-noise ratio declines as model size grows from 167 million to 1.2 billion parameters. Because the MTP branches share the trunk with next-token prediction, the residual can disrupt backbone training while obscuring the distillation signal needed by the draft heads.
Recovering KL without dense soft targets
Direct KL minimization would use the backbone’s next-token distribution as the teacher and the MTP distribution as the student. A straightforward implementation must operate on full vocabulary distributions for both branches, increasing memory traffic and losing compatibility with fused hard-label cross-entropy kernels used by many training stacks.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
