What's this about?
Researchers digging into Inkling's technical details are excited about GRAPE, Thinking Machines' new "query-gated" attention design. The claim: it replaces several standard tricks at once — RoPE positional encoding, attention sinks, and gated attention — with one cleaner mechanism. One researcher's verdict went viral: "RoPE is dead, long live GRAPE!"
What exactly is "attention" (and RoPE)?
Attention is the core operation in a transformer: for each word it's producing, the model decides which earlier words to "look at." RoPE (rotary position embedding) is the near-universal trick for telling the model *where* each word sits in the sequence — almost every modern LLM uses it. A lab shipping a trillion-scale model *without* RoPE is like a car company shipping a production car without a conventional transmission: proof the alternative works at scale, not just in a paper.
What's the community saying?
- Architecture nerds are thrilled — open weights mean people can verify the design instead of trusting a paper.
- Long-context implications — killing attention sinks + 1M context suggests GRAPE is central to how Inkling handles long inputs.
- Will others adopt it? Speculation that open labs will copy the recipe in their next training runs, the way RoPE itself spread.