What's this about?
Frontier open-weight models keep being released, but serving them still assumes datacenter infrastructure. FreeToken (61 upvotes) treats a personal machine not as a small GPU but as a unified, elastic inference platform, co-designing the whole serving stack — model layout and loading, expert residency, CPU–GPU execution, agentic state reuse, runtime memory management. Rather than committing to a fixed offloading strategy, it continuously maps computation onto whatever resources are actually available. The results: a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on a single workstation GPU.
What exactly is "expert residency," and why does MoE make this possible?
A mixture-of-experts model contains many specialist sub-networks but activates only a few per token. That means most of the model is idle at any moment — so it doesn't all need to be in fast GPU memory simultaneously. Expert residency is the decision of which experts to keep on the GPU versus in system RAM or on disk, and it's a prediction problem: guess right and inference is fast, guess wrong and you stall waiting on a transfer. Doing this adaptively, based on what the current workload actually uses, is what lets a 753B model run on hardware that cannot possibly hold it in VRAM.
Why is this worth noticing?
- The GLM-5.2 result is the striking one. A 753-billion-parameter frontier model on a single workstation GPU changes what "you need a datacenter" means.
- It's the technical answer to today's Luna delisting. If cheap hosted inference has hit an economic floor, local serving is where the next reduction comes from — and this is the engineering that makes it feasible.
- The agentic angle matters. It explicitly accounts for agent workloads continuously changing their execution pattern, which is what breaks fixed offloading strategies designed for chat.