What's this about?
A transformer trained from scratch in 1.5 hours on a single RTX 5090, matching or beating many LLMs on the ARC-1 public eval and scoring 7% on ARC-2. It is the third in a series, and the author's stated motivation is that sample efficiency is the most important problem in AI today — with the secondary goal of making iteration cheap enough that poor researchers can do it.
What is ARC, and why is it a good sample-efficiency test?
ARC is a set of visual puzzles where each one has a different rule, demonstrated by a handful of examples. It is a metalearning benchmark: you cannot memorise the rules, you must infer each from two or three demonstrations. There are only about a thousand puzzles, in a very high-dimensional space, needing almost no prior knowledge — every concept in the eval appears in the training set. Humans find it easy. Models find it hard. That gap is precisely the sample-efficiency gap.
What's the community saying?
- The main objection: is this just benchmaxxing? The model is optimised for one benchmark, which the field usually calls overfitting and tries to avoid.
- The correction that settles it: overfitting means training on the answers, and he does not use labels — the model does not know them. Scope is limited, but it is not memorising.
- A pointed defence: there is a $700k prize for the next iteration of this benchmark, so "finetuning for a specific task where you have a benchmark" is exactly the game.
- The framing several found most interesting: this is a response to LLM hype, showing computers can solve these problems without an LLM architecture — a reminder that machine learning is more than LLMs.
- A pedantic but correct sub-thread: he built a transformer, not an LLM, and the distinction is the whole point.