What's this about?
TurboKV is a Rust key-value store pitched on raw speed (HN 180 points). vLLM shipped v0.28.0 (HN 107 points), the widely-used open-source inference server for running LLMs efficiently.
What does an inference server actually do?
It sits between your request and the GPU and solves the scheduling problem: batching many users' requests together so the GPU is never idle, managing the key-value cache that stores attention state so tokens are not recomputed, and streaming output back. Running a model without one wastes most of your hardware. vLLM's contribution was PagedAttention — treating the KV cache like virtual memory pages — and it is why self-hosting a large model is practical at all.
Why is it worth noticing?
- vLLM releases are the practical gate on whether new open-weight models like GLM-5.3 and Hy4 are usable outside their vendors' APIs.
- The KV cache is the same object that Codex's compaction bug and the cache-pricing discussion both concern — it is the quiet centre of LLM economics.
- Both are ordinary infrastructure releases; included because open-weight releases are worthless without them.