Geoffrey Hinton, Joshua Bengio, Leo Gao, Stephen Wolfram and Dario Amodei have at least two things in common. All of them, along with many other scientists, have contributed to the advancement of artificial intelligence (AI). Current level of development. And furthermore, they all openly admit that they do not fully understand how language models are internally organized and use what they have learned during their training. In some ways, they behave like black boxes whose contents are relatively inaccessible.
It has been several years since these two scientists made this statement, but recently Sam Altman, Dario Amodei and other relevant figures in the AI industry have acknowledged that large language models have limitations in understanding the inner workings. Nick Camarata is determined to solve this huge problem. This researcher worked at OpenAI as part of a team developing techniques to interpret what neurons represent in language models. And he contributed to the field with pioneering work on the visualization of neural circuits.
“I’m so happy to see this from Nick!”
Nick Cammarata currently works as a researcher specializing in the mechanistic explanation of AI at Goodfire, a company dedicated to investigating how AI models work internally. On October 15, an article titled “Neural networks are now transparent” was published on this company’s website (Neural networks are already clear) in which he argues that, in reality, AI is much more intelligent than we think.
Cammarata does not propose a definitive demonstration of the interpretability resolution of large language models, but presents a very ambitious research hypothesis supported by experiments with computer vision networks and language models. Understand its purpose in as much detail as possible How Neural Networks Workand its starting point is the analysis of neuronal polysemy (polysemanticity) as the ability of some neurons to perceive concepts that are apparently unconnected.
Cammarata does not offer a definitive proof of the interpretability resolution of large language models, but he presents a very ambitious research hypothesis.
The dominant explanation so far has adopted superposition, which maintains that a neural network can encode different features in a single neuron because it contains a finite number of them. However, Camerata believes that interpretation is incorrect in most situations. His thesis maintains that neurons do not arbitrarily mix concepts, but rather respond to shared visual patterns. The same neuron, for example, can be activated by both the face of a cat and the front of a car, two objects that apparently have nothing to do with each other, because in both cases it can perceive a V-shaped pattern.
Another interesting idea that Camerata flirts with in his article is the idea of recursive generalization. According to this method, a neuron can start by detecting a pattern and learns the model, Related variants included. A neuron capable of responding to a sequence of letters can identify words and expressions that contain that sequence. Camarta’s proposal is that this grouping is not arbitrary: it is the result of the relationships established by the neural network itself. For this reason, a better understanding of neurons can help us understand how language models form abstractions.
One of the most interesting conclusions that Nick Cammarata reaches in his article is that current neural networks can be much easier to interpret than the first generation. It doesn’t seem reasonable if we keep in mind that currents are more complex, but Cammarata attributes the initial difficulties to design and architectural issues that prevent learning deeper layers, imperfect visualization tools and evaluation methods that can confuse a rational explanation with a correct one.
The response to this researcher’s proposal did not take long. Shortly after his article was published, Sam Altman, his former boss at OpenAI, posted a tweet at X in which he praised Camerata’s work quite eloquently: “I’m so happy to see this from Nick!”. I hope I’m right. Hopefully neural networks will make sense soon. This is the best path we can follow to enhance AI security and ensure proper alignment of frontier models.
Picture | Anthony Sakrba
More information Goodfire
In Xataka | Two invisible settings that decide whether your AI is right or wrong
In Xataka | “We’re already at the AI singularity”: Sam Altman, CEO of OpenAI
