ICML Paper Maps When RLHF Beats DPO and Online DPO Wins
ICML paper decomposes the RLHF vs DPO gap into explicit (exact optimization) and implicit (finite sample) representation gaps. No universal winner: RLHF, DPO, or online DPO can each dominate under different model mis-specifications.
Takeaways
- ICML paper decomposes the RLHF vs DPO gap into explicit (exact optimization) and implicit (finite sample) representation gaps.
- No universal winner: RLHF, DPO, or online DPO can each dominate under different model mis-specifications.
- Online DPO beats both when reward and policy classes are isomorphic and both imperfect, the typical LLM regime.
- RLHF needs far fewer samples than DPO when the true reward is sparse or structured.
- DPO wins when the policy class is strictly more expressive than any reasonable reward model.
- Theory only, no code, Bradley-Terry assumption throughout. Paper link .
Developers choosing among RLHF, offline DPO, and online DPO often compare end-to-end benchmark results that entangle model capacity, preference coverage, and optimization behavior. A paper accepted at ICML isolates the effects of model misspecification and finite preference data. Its analysis gives each method a path to outperform the others, helping teams decide where to place modeling assumptions and spend labeling resources.
One objective, three training routes
RLHF first fits a reward model to pairs of chosen and rejected responses, then optimizes a policy against that learned reward. Practical systems often use PPO for the policy update. DPO trains the policy directly on preference pairs, avoiding a separately optimized reward model. Online DPO repeats that direct update while collecting new comparisons from responses generated by the current policy.
For KL-regularized reward maximization, the optimal policy follows π*(y|x) ∝ π_ref(y|x) exp(r(x,y)/β) . The reference policy constrains drift, and β controls the strength of that constraint. DPO substitutes this reward-policy relationship into the preference likelihood and optimizes the policy parameters directly. The paper assumes a Bradley-Terry preference model, which converts the reward difference between two responses into the probability that one is preferred.
| Method | What it fits | Preference source | Potential advantage |
|---|---|---|---|
| RLHF | An explicit reward model, then a policy | Usually a fixed dataset | Can encode reward structure directly |
| Offline DPO | A policy through preference likelihood | A fixed dataset | Uses policy expressiveness without a separate reward model |
| Online DPO |
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
