What's this about?
Sparse autoencoders extract large numbers of interpretable features from a language model's internal representations, but explaining what each feature *means* still relies mainly on external observation — looking at which inputs activate it and guessing. This paper argues that produces superficial explanations, and proposes having the representation verbalize itself.
What exactly is a sparse autoencoder?
Inside a model, individual neurons are frustratingly polysemantic — one may activate for Python code, legal language, and the colour blue, because the model packs more concepts than it has dimensions. A sparse autoencoder re-expresses that jumbled internal state as a much larger set of features where only a few are active at a time, which tends to separate the tangled concepts into individually meaningful ones. It's currently the most promising tool for reading a model's internals, which makes explaining the resulting features the immediate bottleneck.
Why is this worth noticing?
- Sparse autoencoders produce millions of features and labeling them by hand doesn't scale, so automating the explanation step is what determines whether the technique is usable at all.
- Asking the representation to explain itself rather than inferring from the outside is methodologically interesting — and carries an obvious risk, since a model's account of its own internals is exactly the kind of claim that needs independent verification.