What's this about?
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, which turns the state transition into an online learning rule. The paper studies that rule under read-after-write autoregressive semantics. 20 upvotes.
What is a "fast weight"?
Standard weights are learned during training and frozen at inference. Fast weights change *during* inference, as the model reads — a small, temporary memory written into the network's parameters rather than into a growing context buffer. The appeal is constant memory cost regardless of sequence length. The catch is that a fixed-size state must forget something to learn something, so the state transition becomes an implicit decision about what matters, made continuously and irreversibly.
Why does this matter to you?
- It is the theoretical end of the same problem LayerRecall and the compaction work approach practically.
- Fixed-size state versus growing context is one of the genuine architectural forks in front of the field.
- "Continual learning" here means within a sequence, not across deployments — a distinction the term routinely blurs.