What's this about?
Researcher Matt Henderson visualized what a multimodal LLM "thinks" while watching a video: Gemma 4 12B consumes video as raw image patches in the same stream as text tokens, and although it was never trained to predict *at* those positions, you can still sample its next-token head there — producing an eerie frame-by-frame readout of what the model would predict next.
What exactly is an "image patch"?
Multimodal models don't see pictures; they see patches — the image chopped into a grid of small squares, each converted into a token-like vector and fed into the same sequence as words. Gemma 4's design ("unified, encoder-free") skips the separate vision encoder older models used, treating patches almost exactly like text tokens. That's what makes this visualization possible at all: the prediction head can be pointed at patch positions as if they were words to complete.
What's the community saying?
- Interpretability as spectacle — a rare visualization that makes the internals of a multimodal model legible (and shareable) to non-researchers.
- Encoder-free architecture validation — the demo doubles as an argument for the unified-token design Gemma 4 introduced.
- "What else can we sample?" — researchers riffing on probing other never-trained positions to see what latent predictions models carry.