What's this about?
A Rust gateway (rewritten from an initial Python version) that sits between you and multiple model providers and uses your own usage data to decide which model handles which request. The most interesting thing in the thread is the author undercutting their own pitch.
What is a "model gateway"?
A single API endpoint that fronts many models. Your code calls one place; the gateway decides whether that request goes to a cheap fast model or an expensive careful one. The appeal is obvious — you stop hard-coding model names — and the hidden cost is equally real, which the thread surfaces immediately.
What's the community saying?
HN 117 points, 17 comments
- The first substantive question is about prompt caching: sticking with one model saves a lot of money on cached input tokens, and swapping models throws that away.
- The author's answer is the honest one — switch rarely, or only at task boundaries — and then goes further: "often the conclusion of routing is actually 'this one model is at the Pareto front for this task, just use it always'."
- Which prompts the obvious follow-up: if that's true, don't put a gateway in the path at all, just let the harness pick models for its sub-agents.
- The author's remaining claim is maintenance: automatically figuring out which models sub-agents should use, and updating that as new models ship.