APIUp to 28% cheaper than official pricesTry the API →
Hermes
Back to News

Goodfire Finds Tiny Neuron Teams Inside Llama 3.1 8B Steering Grammar

Goodfire's Neuron Communities method finds thousands of small neuron groups inside Llama-3.1-8B without supervision. Communities, often under ten neurons, are causally necessary and sufficient for behaviors like grammar, arithmetic, copying and formatting.

Goodfire Finds Tiny Neuron Teams Inside Llama 3.1 8B Steering Grammar
Source
Goodfire
Published
Author
Goodfire
Read
6 min read

Takeaways

  • Goodfire's Neuron Communities method finds thousands of small neuron groups inside Llama-3.1-8B without supervision.
  • Communities, often under ten neurons, are causally necessary and sufficient for behaviors like grammar, arithmetic, copying and formatting.
  • Pipeline: gradient attribution on 10M prompts, a neuron co-occurrence graph, then Leiden community detection.
  • Coverage is 10.3% of held-out prompts, against a 3% baseline for size-matched random neuron sets.
  • Hundreds of thousands of communities were also found in Qwen 3.6 35B-A3B, including fact-storing circuits.
  • Paper , code and the explorer are free. Robustness on off-distribution inputs is untested.

Goodfire has found thousands of recurring neuron groups in Llama 3.1 8B by analyzing ordinary text without task labels. In one group, six MLP neurons spread across six transformer layers steer the model after it emits “an,” raising the probability of a word that begins with a vowel. Ablating those neurons produces errors such as “an new.” Zeroing every other MLP neuron in the six layers largely preserves the original prediction when the six-neuron group and four ubiquitous background neurons remain active.

The project calls these groups neuron communities . Research fellow Igor Shilov of Goodfire and Imperial College London led the work with Thomas Fel, Tal Haklay, Lucius Bushnaq, and Owen Lewis. Goodfire published a research report , an arXiv paper, source code, and an interactive explorer. All are freely available.

Neurons get a second look

Interpretability researchers have often treated individual neurons as poor units of explanation. A neuron can be polysemantic, activating in several apparently unrelated contexts, while a single prompt can activate thousands of neurons. Activity alone therefore provides little evidence that a neuron causes a behavior.

Sparse autoencoders and transcoders address that problem with auxiliary networks trained to recode model activations as cleaner, sparse features. Those learned features do not map one-to-one onto the model’s actual neurons. Goodfire’s companion study found that direct neuron analysis may still expose compact computations. The team zeroed the weakest 92.2% of neuron activations in Qwen3.6-35B-A3B, increasing next-token prediction loss by 2.9%. An earlier “Arithmetic in the Wild” paper identified 28 neurons in Llama 8B involved in base-10 addition. Ablating them reduced accuracy from 95% to 24%, while retaining only those neurons in their layer preserved 86% accuracy.

Those results motivated a broader test: whether small neuron groups recur across inputs as stable computational units. Neuron communities are Goodfire’s method for finding and testing those groups automatically.

Ten million prompts become a graph

Goodfire builds candidate communities without specifying a target behavior such as addition or grammar. The pipeline has three stages:

  1. Attribute predictions. Run Llama 3.1 8B on 10 million prompts from FineWeb-Edu, English Wikipedia, and FineMath. For each next-token prediction, use output gradients to estimate how much each MLP neuron contributed.
  2. Build a co-occurrence graph. Represent each neuron as a node. Connect two neurons when both repeatedly rank as important for the same predictions. Normalize the edge weight by each neuron’s individual frequency so neurons active on nearly every input do not dominate the graph.
  3. Detect communities. Apply the Leiden graph-clustering algorithm to find tightly connected groups whose neurons repeatedly contribute together.

The following Python sketch captures the graph-building idea. NetworkX’s Louvain implementation provides a compact approximation to the paper’s Leiden clustering step.

Ablation separates clusters from circuits

  • Necessity test: Zero the candidate community on held-out prompts and compare the new next-token distribution with the original. A large KL divergence indicates that the intervention changed the model’s output substantially.
  • Sufficiency test: Keep the community active while zeroing every other MLP neuron in the layers it spans. A largely preserved prediction indicates that the group carries much of those layers’ relevant contribution.

The sufficiency intervention also preserves 17 background neurons distributed across the model’s 32 layers because they activate on nearly every input. Four of those neurons occur within the six layers used by the “an” community, which accounts for the four additional neurons retained in that example.

Grammar, arithmetic, formatting, and suppression

  • Grammar: Seventeen neurons across eight layers choose between “is” and “are” after “there.”
  • Retrieval and computation: Communities handle unit conversion, multiplication, and in-context lookup. One seven-neuron group across four layers retrieves names mentioned earlier in a prompt. The method also rediscovered an eight-neuron addition community without an addition-specific search.
  • Formatting: A seven-neuron community continues a phrase in all caps. Other groups close brackets, finish LaTeX commands, or preserve title case.
  • Common phrases: A 37-neuron community predicts “out” in phrasal verbs such as “carry out” and “figure out.”
  • Topics: A three-neuron community activates on trigonometric functions.
  • Token suppression: The most common communities contain one or two neurons that reduce the probability of a specific word.

Community locations provide a rough map of computation across the network. Grammar neurons cluster from layer 18 onward, retrieval and arithmetic peak around layers 17 through 22, and token-suppression communities concentrate in the final layers. The researchers also report hundreds of thousands of communities in Qwen3.6-35B-A3B, including groups associated with facts such as state capitals and membership in the post-Impressionist movement.

Layer-level coverage reaches 10.3%

With community size capped at 128 neurons, 10.3% of held-out prompts have at least one layer whose contribution is explained by a discovered community. Size-matched random neuron groups reach 3%. The comparison shows that the graph procedure finds more causally relevant groups than random selection, while the coverage figure leaves most prompts unexplained.

  • Sufficiency applies only within the layers spanned by a community. Other layers, attention blocks, embeddings, and the rest of the network remain active.
  • Necessity and sufficiency identify causal involvement on tested prompts. They do not reveal the algorithm implemented by the neurons or their interactions with other components.
  • The 10.3% result concerns at least one layer’s contribution to a prompt, not a complete explanation of the model’s prediction.

Bret Kerr’s prepublication critique asked whether neuron-level explanations remain stable on jailbreaks, gibberish, glitch tokens, and very long contexts, where safety monitoring faces difficult inputs. He also requested size-matched random controls, stability across random seeds, and replication across architectures. The paper’s 3% baseline supplies the random control. Stability across seeds and models, along with adversarial and long-context testing, remains open.

From hand-built circuits to a search index

Earlier circuit studies commonly began with a behavior selected by a researcher and traced its path through the model. Goodfire’s pipeline begins with millions of ordinary prompts and produces a catalog of candidate mechanisms. Each entry includes a neuron list, examples that activate it, and causal intervention results. Goodfire plans to give these entries to AI agents that can generate and test hypotheses about each group’s function.

The findings make real MLP neurons practical targets for some interpretability experiments. Small, causally tested groups may reduce the need for sparse autoencoders or transcoders when direct neurons provide enough coverage. The 10.3% coverage and the focus on MLP computation keep that conclusion narrow.

Goodfire has also announced an embedded assessment with OpenAI. Its researchers will monitor training runs and investigate unwanted behavior with the company’s interpretability tools, providing a commercial setting for the same approach.

Three immediate uses

  • Circuit research: The explorer supplies tested starting points for studying Llama 3.1 8B without running the full discovery pipeline first.
  • Debugging and editing experiments: A named group of neurons offers a smaller intervention target than an entire layer, though changes still require evaluation across unrelated prompts.
  • Teaching: Examples such as the “an” community demonstrate necessity and sufficiency tests in a production-scale language model.

Open resources and compute costs

The community explorer exposes the discovered groups and their activating prompts. The arXiv paper provides the experimental details, while the GitHub repository contains the pipeline for running the method on another model.

Reproducing the published results requires attribution over 10 million prompts, making that stage the main computational expense. The current evidence supports using communities as research leads and intervention targets. Production safety monitoring still requires robustness tests on adversarial inputs, long contexts, random seeds, and additional model families.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Sources

  1. 01x.com
  2. 02goodfire.com
  3. 03arxiv.org
  4. 04arxivsignals.io
  5. 05goodfire.com
  6. 06fourweekmba.com