What's this about?
A Show HN running the 104GB Qwen3.8-Flash-Next on a 48GB Mac at around 12 tokens per second, by keeping the resident parts of the mixture-of-experts model in unified memory and streaming the routed experts from SSD.
Why does a mixture-of-experts model make this possible?
Because MoE only activates a fraction of its parameters per token. The model is 104GB in total, but any given token routes through a small subset of experts. So you keep the always-needed parts resident and page the rest from disk on demand. It works because SSDs are fast and the routing is sparse; it produces 12 tokens per second rather than the hundreds you would get with everything resident, which is slow but is the difference between running it and not.
What's the community saying?
HN 147 points, 90 comments
- The top comment lists five existing projects doing essentially the same thing and asks people to collaborate on one rather than each wrapping the idea in a new README.
- The author responds well — he did do the research, will add a comparison table, and notes this is not commercial at all.
- The defence that gets upvoted: "Does a painter check to make sure that a portrait hasn't been painted?"
- Underneath is a real ecosystem observation: this is open source since forever, and half of HN is running Ollama without doing market research either.