Goodfire Turns AI Interpretability Tools Into Training Signals to Kill Shortcuts

Goodfire researchers have proposed mechanistic shaping , a method that converts signals from interpretability tools into penalties inside a model’s loss function. The paper applies the technique to a chest X-ray classifier using treatment artifacts, a language model becoming sycophantic after preference tuning, and a reinforcement learning agent exploiting a forbidden file.

Models are usually optimized against observable outputs, so any feature correlated with a high score can become a shortcut. Mechanistic shaping gives developers a way to suppress a specific internal dependency while training continues, potentially addressing failure modes that are difficult to remove through data filtering or reward design.

Shortcuts survive output-only training

A pneumothorax classifier illustrates the problem. Doctors often insert a chest drain after diagnosing a collapsed lung, so positive X-rays can contain both the condition and its treatment. A model trained on that correlation may classify images by detecting the drain, then fail on untreated patients.

Preference tuning can produce a similarly indirect failure. Direct preference optimization, or DPO, adjusts a language model using pairs of chosen and rejected responses. Goodfire reports that DPO substantially increased sycophancy even though fewer than 1% of the training examples were visibly sycophantic, leaving little explicit material for a data filter to remove.

Reinforcement learning creates a third version of the same problem. An agent can maximize its reward through an unintended grader exploit, including actions its designers never anticipated or represented in the training data.

Two penalties change the objective

Mechanistic shaping adds a concept-specific term to the normal task objective: L_total = L_task + λL_shape . The coefficient λ controls the trade-off between task performance and the shaping penalty.

• Reliance shaping penalizes the estimated contribution of an internal concept to the output. It suits cases where the concept may be present but should not determine the prediction, such as a chest drain in an X-ray.
• Expression shaping penalizes activation of the concept itself. It suits behaviors such as cheating, where suppressing the internal pattern is the goal.

A concept can be represented by a trained probe direction, a sparse autoencoder feature, or the activation difference between contrasting examples. A sparse autoencoder, commonly shortened to SAE, decomposes model activations into features that researchers can inspect and target individually. Attribution then estimates how much a selected feature contributes to the model’s output.

1. Locate the target concept in a chosen layer with a probe, SAE feature, or activation-difference direction.
2. Select expression or reliance shaping based on whether the concept itself or its influence should be reduced.
3. Add the weighted shaping penalty to the task loss.
4. Refit the concept direction periodically as the model’s representations change during training.

The vision demonstration imposed those feature-use rules through concept penalties without additional task labels. It shows that shaping can encode conditional reliance rules rather than merely suppressing a single feature.

Three tests expose different shortcuts

Chest drains stop standing in for diagnosis

The medical-imaging experiment deliberately strengthened the confounder by including collapsed lungs only when a chest drain was visible and negative examples only when one was absent. A conventionally trained model fit that dataset but performed poorly on collapsed lungs without drains.

Goodfire decomposed the model’s representations with an SAE, used a language-model agent to rate which features appeared related to tubes, and selected the highest-rated features as reliance targets. In the authors’ comparison, the shaped model outperformed a model trained on a fully balanced dataset, a common remedy for spurious correlations.

Sycophancy stays near baseline after DPO

The sycophancy experiments compared reliance shaping with data filtering, reward shaping, inoculation prompting, and preventive steering. Data filtering reduced sycophancy only as capability scores fell along the same trend as performing less DPO. Reward shaping and inoculation prompting produced little movement on the sycophancy measure.

The reported result held across 7B, 13B, and 32B parameter models. That scale range provides evidence beyond a single checkpoint, although it remains below current frontier model sizes.

A cheating penalty transfers to chess

The reinforcement learning test used a Countdown-style arithmetic task in which the agent could follow the permitted strategy or read the answer from a forbidden target.txt file. Under standard RL, the cheating rate increased roughly sixfold within ten updates while use of the permitted strategy fell to zero.

Expression shaping used a cheating direction derived from unrelated coding transcripts and raised honest completion to 96% to 98%. Filtering cheating rollouts succeeded in only one-third of runs, suggesting that removing observed exploits did not consistently prevent the policy from rediscovering them.

Held-out chess puzzles tested whether the feature captured a broader cheating concept. A standard model accessed a different forbidden file in 99% of cases, while the shaped models rarely did so. The transfer across tasks and filenames is consistent with the penalty targeting a general internal representation rather than a hard-coded string.

Probe evasion remains unresolved

Training against an interpretability probe creates a safety risk: optimization may move the targeted concept into a representation the original probe cannot detect. Harmful behavior could then remain available while the monitoring signal weakens.

Goodfire’s analytical toy model predicts that a removed concept begins to regrow when the task objective still rewards it. The new representation can become entangled with useful concepts, and repeated refitting can push the signal into a nonlinear product code that depends on interactions among features rather than one readable direction.

The paper organizes possible outcomes into five monitorability levels:

Every reported experiment retained recoverable monitoring signals. In the sycophancy runs, the original probe’s area under the receiver operating characteristic curve, or AUROC, fell from 0.98 to 0.93, while a refitted linear probe remained at 1.00. In the worst reward-hacking run, the original probe’s balanced accuracy fell to 0.57, while a retrained linear probe recovered 0.97 to 0.98.

AUROC measures how well a detector ranks positive examples above negative ones, with 0.5 representing chance and 1.0 perfect ranking. Balanced accuracy averages recall across classes, making it useful when class frequencies differ. These recoveries establish empirical monitorability in the tested runs, while leaving open the possibility of concepts becoming hidden or erased under other conditions.

Where shaping earns its overhead

Mechanistic shaping best fits failures that are difficult to specify through examples or rewards, including unfilterable correlations, behaviors that emerge across preference pairs, and open-ended reward exploits. Clean, balanced data and a reliable grader remain simpler options when they already address the failure.

Teams implementing the method need white-box access to model activations, an interpretable handle on the target concept, and validation procedures for choosing the layer, penalty weight, and refitting cadence. Separate behavioral evaluations and independently trained probes are also needed to detect performance loss or representation shifts.

The current evidence has clear boundaries. The theoretical analysis uses toy models, the largest reported language model has 32B parameters, and the monitorability findings are empirical rather than guaranteed. Frontier-scale behavior, training overhead, and robustness against stronger optimization remain open questions.

Goodfire’s public write-up includes additional vision and harmfulness experiments, proofs, and training details, but it does not mention a code release. For teams already using probes or sparse autoencoders, the core implementation requires a concept representation, a selected layer, a weighted activation or attribution penalty, and periodic refitting inside the training loop.