APIUp to 25% cheaper than official pricesTry the API →
Hermes
APIModelsCases
Back to News
paperTraining InfraLlms

Synthetic Warm-Up Saves 21B Training Tokens by Boosting Long-Range Retrieval

The largest study of synthetic pre-pretraining shows it saves 21B+ tokens at 3B scale, but the grammatical prior story falls apart.

Synthetic Warm-Up Saves 21B Training Tokens by Boosting Long-Range Retrieval
Source
AlphaSignal
Published
Author
AlphaSignal Newsroom
Read
1 min read

The largest study of synthetic pre-pretraining shows it saves 21B+ tokens at 3B scale, but the grammatical prior story falls apart.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Next reads

Sewon Min's Preprint Shows Smaller Batches Beat Overfitting on Repeated Training Data
AlphaSignal · paper

Sewon Min's Preprint Shows Smaller Batches Beat Overfitting on Repeated Training Data

MSAR Cuts Transformer Training Compute by 37% With Smarter Residual Connections
AlphaSignal · paper

MSAR Cuts Transformer Training Compute by 37% With Smarter Residual Connections

Zyphra Cuts MoE Training Communication Costs by 2.63x Using Router Patterns
AlphaSignal · paper

Zyphra Cuts MoE Training Communication Costs by 2.63x Using Router Patterns