What's this about?
Scaling laws for compute-matched mixture-of-experts looped transformers. 77 upvotes.
What is a looped transformer?
A model that runs the same layers repeatedly instead of stacking more distinct ones — depth through iteration rather than parameters. It gives you more computation per token without more weights to store, which is attractive when memory is the constraint and memory is currently expensive. Combining that with mixture-of-experts, where only a fraction of parameters activate per token, gives two independent dials on the compute-versus-memory trade. "Compute-matched" means the comparisons hold total computation fixed, which is the only way to say anything meaningful about which architecture is better.
Why is it worth noticing?
- Scaling-law papers are how architectural choices get settled, and compute-matched comparison is the right methodology.
- Memory prices make weight count matter more than it did a year ago — see SK Hynix's four-year shortage forecast.
- It pairs with yesterday's "sliding-window beats linear attention" as architecture work aimed at the same constraint.