What's this about?
SWE-Bench ProMax is a new, harder benchmark for coding agents — this time on large-scale, multilingual code refactoring, not just single-file bug fixes. It raises the bar as the original SWE-Bench gets saturated.
What exactly is a "coding-agent benchmark"?
SWE-Bench is the standard test where an AI agent is given a real GitHub issue and must produce a working code patch that passes the repo's tests. It's how labs claim "our agent solves X% of real issues." As models climb the scoreboard, benchmarks have to get harder to stay meaningful — hence a "ProMax" tier with bigger, cross-language refactors.
Why is this notable?
Refactoring across a large multilingual codebase is much closer to real engineering than fixing one file, so scores here are a better proxy for whether coding agents can handle production work.