Princeton and UC Berkeley's Value Forcing Beats Flow Critics Across 26 Tasks

Value functions remain difficult to estimate when demonstrations are limited and rewards arrive late. Researchers at Princeton and UC Berkeley address that failure in the Value Forcing paper , which introduces a Bellman objective for flow-based critics and reports gains across 26 long-horizon offline reinforcement learning tasks.

Small datasets amplify bootstrap errors

An offline RL transition records an immediate reward, while the total future return remains unobserved. Temporal-difference methods estimate that return from their own predictions, allowing early errors to propagate through long trajectories. Sparse rewards and limited data leave few observations to correct those errors.

Scalar critics also discard information by compressing every possible return into an average. An action that produces either -1 or +1 with equal probability has the same mean as an action that always produces 0, despite carrying a different outcome distribution.

Distributional RL preserves that uncertainty by modeling the full return distribution. Recent critics have used flow matching , a generative method that learns a velocity field for transforming Gaussian noise into return samples through an ordinary differential equation, or ODE.

The noise schedule exposes the failure

The paper traces a recurring problem in earlier flow critics to their Bellman supervision. Their learned return distributions often contract into narrow peaks, erasing uncertainty and behaving much like scalar estimates.