A Former OpenAI Researcher Just Changed His Mind About the AI Black Box

On October 15, Nick Cammarata posted on X. He used to work on mechanistic interpretability at OpenAI. He now works at Goodfire. The post pointed to a long essay he had just published: "Neural Networks Are Already Clean." The essay argues that the internal structure of large neural networks may be far easier to understand than researchers have assumed. The claim drew attention because Cammarata spent years inside the field he is now revising. He helped build tools that used large language models to explain neurons. He once believed that complex networks were tangled with feature overlap. Now he says the opposite.

Mechanistic interpretability tries to map the neurons and circuits inside an AI model. The goal is to trace how an input becomes an output. If researchers can follow that path, they can find errors, bias, and deceptive behavior. In early 2025, MIT Technology Review listed mechanistic interpretability as one of the "10 Breakthrough Technologies" for 2026. The work supports AI safety and alignment. But for years, the field has been stuck on a simple problem: neurons do not behave like clean switches. One neuron may fire for cats, cars, and a dozen unrelated things. As models grow larger, the mess seems to grow with them. Researchers built extra tools to pull apart mixed signals. They trained sparse autoencoders to decompose activations into named features. Anthropic and Google DeepMind put large teams on that work.

The most cited example came from OpenAI in 2023. A team used GPT-4 to watch GPT-2. The larger model generated natural-language explanations for smaller neurons, then tested whether those explanations predicted activation. The result was weak. Even with a loose standard, the tool explained about 1.7% of neurons reliably. Most neurons looked like they handled several unrelated concepts at once. That result helped push the "superposition" hypothesis into the center of the field. The idea is simple: a network has limited physical neurons but must represent a huge number of real-world features. To save space, it stores unrelated concepts in overlapping directions. A single neuron responds to many things because it is carrying too much. Sparse autoencoders became the standard fix. Researchers trained auxiliary models to separate mixed activations into cleaner features.

Cammarata now says the field may have accepted superposition too quickly. In a first pass over Qwen3-32B, his team estimated that about 90% of neuron response directions match one clear dominant function. Another 7% match two closely related functions. Only 2% to 4% still lack a clear pattern. The team says it saw similar numbers in trillion-parameter models, including Kimi K3. These numbers cannot be compared directly with the 2023 result. The models, tools, and evaluation standards are different. The definition of a "single function" also changes the count. But the gap between 1.7% and 90% is large enough to force a question. Were researchers looking at the wrong thing?

The cat-face and car-front case is the clearest example. In the early vision model InceptionV1, one neuron fired strongly for both cat faces and car fronts. Researchers used that neuron for years to show that neural networks mix unrelated concepts. Cammarata re-examined the activation samples and the connections to upstream and downstream neurons. He found a shared feature that earlier work missed. The neuron was sensitive to a sharp, outward-opening V-shaped geometry. Fox faces triggered it. Flames and water jets triggered it. Soft, round shapes suppressed it. The neuron was not representing "cat" and "car" as semantic categories. It was tracking a basic visual shape that appears across categories.

Other confusing activations came from the tools themselves. Early visualization methods sometimes added textures and artifacts to generated images. Those artifacts made researchers misread what a neuron detected. After adjusting the visualization and tracing connections, some previously puzzling responses became explainable through shared pose and geometry. Model architecture also shapes what researchers see. Cammarata retrained and compared InceptionV1 with ConvNeXt, a modern vision network. With the same training data and similar parameter counts, ConvNeXt's last layer had about 99% of neurons with relatively clear functions. InceptionV1's last layer had long been considered hard to interpret. The older architecture may have been part of the problem.

Cammarata also ran an extreme compression experiment. He cut one layer of InceptionV1 from 480 neurons to 10 and retrained the model. The neurons did not turn into random mixtures of unrelated concepts. They formed broader responses to continuous curves and closed loops. The model generalized. Cammarata calls this process "recursive generalization." When capacity is low, one neuron covers a wider range of features. When capacity grows, neurons split into finer concepts. The same pattern appears in language models. A neuron that responds to the letter pair "iy" may also fire for "diy." It may extend to "do it yourself," which shares meaning but not the same letters. If you only look at the final activation list, the responses look disconnected. If you follow the connections, you can find a path from letters to words to meaning.

The study also changes how the research is done. Cammarata uses AI agents to run parts of the analysis. In 2023, GPT-4 could generate natural-language explanations for neurons, but it mostly summarized existing activation samples. The new agents can observe activation data, propose functional hypotheses, inspect upstream and downstream connections, and design new tests. If a hypothesis fails, the agent looks for counterexamples. Cammarata describes a blind test. One AI agent writes a detailed explanation of a neuron based on its activation patterns. A second, independent agent cannot see the neuron's parameters or activation data. It only reads the explanation and then writes new text that should activate the neuron. In some experiments, the agent-generated text produced stronger activation than any sample found in a corpus of hundreds of billions of tokens.

A good explanation should predict. It should not only explain why a neuron fired in the past. It should predict what new input will make it fire again. In a test of 200 randomly selected neurons, researchers asked AI agents to generate positive samples that should activate the neuron and negative samples that should not. Among explanations that could be evaluated, 76% of the new positive samples caused strong activation. The false activation rate for negative samples was 11%. The team also used AI agents to inspect up to 100,000 attribution graphs. They followed paths of information transfer to see how different computation steps affect the model's output. Cammarata argues the work compounds. As more neurons get explained, researchers can use known connections to infer the parts that remain unknown. The map gets fuller over time.

If the direction holds, the methods of interpretability may shift. For years, researchers spent large amounts of compute and engineering time on sparse autoencoders and other decomposition tools. Cammarata's work suggests that for some networks, researchers can start with raw neurons and their connections. They can use AI agents to run systematic analysis without building a separate feature decomposition pipeline for every study. That could help AI safety and alignment. If researchers can trace the neurons and circuits behind a judgment or a task, they can look for reasoning errors, unusual goals, or deceptive behavior. They can intervene at specific points. During model development, the same techniques could help engineers locate failures and see what training or fine-tuning changed inside the network.

The results are not settled. The long essay has not been peer-reviewed. It has not been independently replicated. The statistics for Qwen and Kimi need a fuller technical paper. Understanding one neuron is not the same as reconstructing the whole model. Complex behavior involves causality and coordination across many neurons and components. The superposition hypothesis still has theoretical and experimental support. This work does not overturn it. It proposes a different reading of the evidence. For now, the claim rests on a blog post, a set of agent-generated tests, and a re-examination of old cases. The next step is replication. Until then, the black box is still a black box. Researchers may have been looking at the wrong signal.