Mechanistic Interpretability: Unraveling the Mysteries of Neural Networks

Mechanistic interpretability aims to understand how neural networks actually work under the hood. Neil Kubler, a researcher at DeepMind, discusses key concepts and recent advances in this emerging field.

What is Mechanistic Interpretability?

Mechanistic interpretability seeks to reverse engineer the internal algorithms and computations happening inside neural networks. Rather than just looking at inputs and outputs, it tries to understand why models make the decisions they do.

Some key principles of mechanistic interpretability according to Kubler:

• It engages with the actual mechanisms and computations learned by models, not just surface-level correlations
• It favors depth over breadth, aiming for rigorous understanding of specific circuits/features
• It has an ambitious vision of fully understanding model internals, even if that's not yet achievable
• It focuses on the residual stream as the central object in transformer models

Kubler argues that mechanistic interpretability is possible in principle because neural networks are ultimately made of linear algebra operations trained to perform tasks. By carefully analyzing their internal structure, we can gain insight into the algorithms they've learned.

Circuits and Features in Neural Networks

A key idea in mechanistic interpretability is that neural networks learn discrete "circuits" or "features" that represent meaningful concepts.

Kubler explains that features are often represented as directions in the high-dimensional space of neuron activations. For example, there might be a "gender direction" and a "royalty direction" that combine to represent concepts like "king" and "queen".

However, features don't necessarily align neatly with individual neurons. This leads to the phenomena of "polysemantic neurons" that seem to respond to multiple unrelated concepts.

Superposition

Superposition is a hypothesis for how neural networks can represent more features than they have neurons or dimensions:

• Features are represented as almost-orthogonal vectors
• This allows packing in exponentially many features
• But it leads to interference between features

Kubler discusses two types of superposition:

1. Representational superposition: Compressing many features into a lower-dimensional space (e.g. residual stream)
2. Computational superposition: Computing new features from existing ones using nonlinearities

He argues superposition is likely happening in real neural networks, based on empirical studies like the "Finding Neurons in a Haystack" paper.

Induction Heads

Induction heads are a circuit discovered in language models that allows them to predict repeated sequences. For example, if "Tim Berners-Lee" appears once in a text, an induction head allows the model to more easily predict "Lee" after seeing "Tim Berners-" again later.

Kubler explains that induction heads:

• Seem universal across different language models
• Underlie more complex behaviors like in-context learning
• Emerge suddenly during training in a phase transition

He argues studying specific circuits like induction heads can give insight into emergent model behaviors.

Implications for AI Alignment and Safety

Kubler believes mechanistic interpretability could be valuable for AI alignment and safety:

• It may allow us to better predict emergent capabilities in AI systems
• It could help us understand if/how models develop goal-directed behavior
• It may enable auditing models for safety/alignment before deployment

He argues we're currently very confused about the internals of large language models, and mechanistic interpretability could reduce this confusion.

However, Kubler cautions against overconfidence, noting there's still a lot of uncertainty around how applicable current techniques will be to future AI systems.

The Future of Mechanistic Interpretability

Kubler is optimistic about the potential of mechanistic interpretability, but notes the field is still very young. He encourages more researchers to get involved, highlighting opportunities to make meaningful contributions.

Some open problems he's excited about:

• Better understanding computational superposition
• Reverse engineering circuits in larger language models
• Developing more rigorous, automated interpretability techniques

Ultimately, Kubler hopes mechanistic interpretability can give us a deeper scientific understanding of neural networks and help address crucial questions around AI capabilities and safety.

While many challenges remain, this emerging field offers a promising approach to peering inside the black box of modern AI systems.

Article created from: https://www.youtube.com/watch?feature=shared&v=_Ygf0GnlwmY