What's this about?
K2's architectural contribution is Mixture-of-Value-Attention (MoVA): extending mixture-of-experts routing — normally applied only to feed-forward layers — into the attention mechanism itself.
What exactly is "mixture of experts"?
Instead of every parameter processing every token, the model routes each token to a small subset of specialist sub-networks — a hospital triaging patients to specialists rather than sending everyone to every doctor. MoVA applies that triage inside attention, the layer that decides what context matters.
Why is this worth noting?
MoE-in-attention has been tried before but rarely shipped at scale in an open model — and because K2 publishes its training logs, this is the first time outsiders can study how such a design trains, not just how it benchmarks.