Transformers Secretly Track Each Agent's Beliefs in Separate Brain Regions

A new preprint reports that transformers trained on sequences from agents linked by feedback encode a separate belief representation for each agent. Linear probes recover those beliefs from distinct regions of the residual stream, and activation patching shows that editing one region changes the model’s next-token prediction in the expected direction.

The representational cost motivates the result. A full joint belief vector must cover every combination of agent states, so its size can grow exponentially with the number of agents. A factorized representation stores one posterior per agent and grows linearly when each agent’s local state space remains fixed. That structure could make multi-agent prediction more efficient while giving interpretability researchers a concrete object to locate and edit.

The object in the residual stream

Computational mechanics describes an optimal predictor through its belief state: the Bayesian posterior over the hidden state of the process generating the next token. For a hidden Markov model, or HMM, that posterior lies on a probability simplex. Repeated Bayesian updates can trace intricate, sometimes fractal sets within the simplex.

An earlier study found that transformers linearly represent these belief geometries in the residual stream, the shared activation vector that passes through successive transformer blocks. Previous results also established factorization for independent and conditionally independent processes. The new work extends the analysis to agents whose outputs feed into one another.

Turning feedback into an HMM

The authors model each agent as a stochastic transducer, a system that updates its internal state and emits an output based on its current state and incoming symbol. A clocked circuit specifies when each output is produced and consumed, resolving the timing ambiguity created by instantaneous feedback.