What's this about?
Zhipu's co-founder published a short history of how the field's understanding of scaling has repeatedly reversed. His frame: every model release ends with "how many parameters?", and that question cannot be answered alone — parameter count only means something alongside how much data you have, where you intend to spend compute, and who will run the model under what conditions. He traces three corrections. Kaplan et al. in 2020 fit an exponent implying parameters should grow faster than data, roughly 2.7:1, and the industry complied — GPT-3, Gopher, MT-NLG. Hoffmann et al. in 2022 redid the experiment across four hundred models and found the compute-optimal ratio is closer to 20 tokens per parameter, with both growing at the same rate. As Tang puts it, "the trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed." Then Chinchilla itself gets corrected: it optimized *training* compute for models trained once, but today inference dominates lifetime cost, which pushes the optimum toward smaller models trained far longer — Llama-2-7B at roughly 290 tokens per parameter, Gemma-2-9B at 889. And sparsity moves it again: in a mixture-of-experts model, total parameters govern how much it can hold, while activated parameters and effective depth govern how far it can think.
What exactly is a "scaling law"?
It's an empirical formula predicting how much better a model gets as you add parameters, data or compute. The reason they matter is economic: before spending tens of millions on a training run, you want to know how to divide the budget between a bigger model and more data. A scaling law claims to tell you the optimal split. The reason Tang's history is uncomfortable is that these laws are fitted from experiments, and a fitting error compounds with every order of magnitude — which is exactly why, as he notes, the largest models of the Kaplan generation were the *most* misallocated. The law wasn't wrong at the scale it was measured; it was wrong where everyone extrapolated it to.
Why is this worth noticing?
- The MoE distinction is the practically useful part. "Total parameters govern how much it can hold; activated parameters and effective depth govern how far it can think" explains why a model can be enormous and still fail a twenty-step reasoning chain — and why finding security vulnerabilities depends more on depth than on how many databases it memorized.
- The task-dependence result undercuts the whole framing. Citing Roberts et al., optimal tokens-per-parameter varies by task — memorization favours more parameters, reasoning favours more data — so there is no single right ratio to publish.
- It's the theory behind this week's GLM-5.3 story. Zhipu improved coding by 50% through post-training alone on the same 743B base with ~40B activated, which is what "scaling is not only of parameters" looks like when someone actually ships it.