Coding benchmarks are an agent story
SWE-bench Verified measures something specific: given a real GitHub issue and the relevant repository, can a model locate the bug, write a fix, and pass the test suite? In early 2024, the best scaffolded systems resolved roughly 12–18% of issues. By 2025, frontier models were past 70%.
| Period | System | SWE-bench Verified |
|---|---|---|
| Early 2024 | SWE-agent + GPT-4 Turbo | ~18% |
| Mid 2024 | Claude 3.5 Sonnet | ~49% |
| Early 2025 | Claude 3.7 Sonnet | ~62% |
| Mid 2025 | Claude Sonnet 4, OpenAI o3 | ~72% |
The standard reading of that trajectory: a developer tools story. Better Cursor, better Claude Code, better Copilot. More pull requests closed without a human touching the keyboard. That reading is correct.
It’s also the smaller of the two effects.
What the benchmark is actually measuring
SWE-bench isn’t testing whether a model can type Python. It’s testing whether a model can take an ambiguous problem description, reason about an unfamiliar codebase, form a plan, handle edge cases, interpret test failures, and revise. The coding benchmark is partly a reasoning and planning benchmark dressed in a software context.
Google pointed at something related early on. They called it “implicit code execution”: when answering questions involving arithmetic, parsing, or structured transformations, the model would quietly write and run Python rather than reason through language alone. The goal wasn’t to make the model a better programmer. It was to make it more reliable at the category of problems language models consistently fail at — anything where an exact answer matters and approximation isn’t good enough.
That was a contained observation about math and string manipulation. It now looks like the design principle for how agents work.
Code as the most general tool
Modern model interfaces don’t arrive as chat boxes. They ship with a standard toolkit: code execution, file access, web search, API calls, browser control, computer use, and extensible connectors through protocols like MCP. The model’s job is to decide when language is sufficient and when to reach for an external system.
Code is the most general of those tools.
A user asks an agent to summarize a 200-row dataset. The agent writes pandas. Asked to generate a formatted report, it reaches for python-docx. Pull live data from three APIs and reconcile the responses? A few lines of requests. The user never asked for code. The user never sees it. It’s infrastructure — the model’s way of offloading work it shouldn’t attempt in language.
Probabilistic work stays in the network: planning, reasoning, language, judgment. Deterministic work gets offloaded to code. The boundary between them is becoming invisible to the user.
That division isn’t a workaround for model limitations. It’s a sensible architecture. Language models are genuinely capable at planning, synthesis, taste, and judgment. They’re genuinely unreliable at arithmetic, parsing, IO, and any task where “approximately right” fails. Code execution handles the second category exactly. Put the two together and the combined system is more trustworthy than either alone.
Why coding gains compound
When a model gets better at coding, it doesn’t only get better at writing functions. It gets better at writing the small, reliable programs that make everything else it does more accurate.
A model that can write a precise data transformation is a more reliable analyst. One that can write a verification script can check its own outputs before returning an answer. One that can compose API calls on demand can pull live context rather than relying on what it already knows. Each improvement in code quality ripples outward into every agentic task that touches files, data, or external systems.
The SWE-bench number matters for developers using agentic IDEs. It matters more for anyone building agents that operate over data pipelines, documents, APIs, and automated workflows — which is most of what agents are actually being deployed to do.
So when a frontier lab ships a model with a meaningfully higher SWE-bench score, the developer story is the visible one. The agent story is the one worth tracking.
Better coding doesn’t just mean better software. It means a more capable and reliable agent for anything deterministic — and there is considerably more deterministic work in the world than the benchmark headline suggests.