AI Coding Agent ROI: Measuring Past the 24% Pull Request Lift

The AI coding productivity literature in 2026 looks contradictory. Microsoft's field study found adopters merging around 24% more pull requests. DX Research, surveying roughly 121,000 developers, found productivity gains plateauing near 10% despite 93% adoption. Other published results span from a 20% slowdown to a 100% speedup.

The instinct is to decide which study is right. That is the wrong move. They are measuring different things over different windows, and the gap between them is the most commercially useful finding in the whole literature.

Why credible studies disagree

Four variables account for most of the spread.

What was measured. Time to complete a defined task is not the same as pull requests merged, which is not the same as features delivered. Each is further from the developer's keyboard and closer to business value, and the measured gain shrinks at each step.

Who was measured. Gains differ sharply between developers new to a codebase and developers who know it deeply. The most-cited counterintuitive result in this space came from experienced open-source maintainers working in code they knew intimately, where AI assistance slowed them down — plausibly because the overhead of reviewing generated code exceeded the cost of writing it themselves.

Codebase conditions. Clean, well-typed, well-tested code with clear conventions is a good environment for a coding agent. A large legacy codebase with implicit conventions and thin test coverage is not. The same tool produces very different results across the two.

The measurement window. This is the decisive one. Measure the week of the merge and you capture the gain. Measure the following month and you capture the rework, review burden and incidents that the gain partly financed.

The costs that arrive late

The 2026 analyses converge on a pattern worth taking seriously: immediate task-speed gains often carry downstream costs that surface weeks after merge.

Rework. Code accepted quickly but not quite right gets revised later. This appears in a metric nobody was watching, some distance from the pull request that produced the headline number.

Review load. When generation stops being the bottleneck, review becomes it. Reviewers face more code, and reviewing generated code is a different cognitive task from reviewing a colleague's — it lacks the shared context that lets a reviewer skim safely.

Incidents. Some quality regression reaches production. Attributing incidents back to AI-assisted code is methodologically awkward but not impossible if you tag provenance at merge.

Token spend. At organisational scale this becomes a genuine line item, reaching millions of dollars annually in large deployments. It also scales with adoption rather than headcount, which makes it behave differently from seat-based tooling budgets.

Tool brittleness. A 2026 analysis categorising more than 3,800 publicly reported bugs across Claude Code, Codex CLI and Gemini CLI concluded that current coding agents remain fairly brittle. Engineering time absorbing that brittleness is a real cost even when it never reaches production.

The four metrics worth tracking

A defensible ROI picture needs four numbers together. Any one alone will mislead.

Metric	What it captures	Why alone it misleads
Merged output	The headline gain	Ignores whether output was correct
Rework rate	Code changed again soon after merge	Noisy without a baseline
Incident rate	Quality reaching production	Lags by weeks
Token spend	Direct marginal cost	Says nothing about value

Rework rate is the one most teams lack and most need. Define it simply — lines or files modified again within some window after merge — and establish the baseline before rolling out, because without a pre-adoption baseline the number is uninterpretable.

Measuring honestly

The methodological trap in this area is that the people evaluating the tools usually want them to succeed. Guarding against that requires a little discipline.

Establish baselines before rollout, not after. Retrospective baselines reconstructed from memory or from a period that already included partial adoption are close to worthless.

Compare cohorts rather than time periods where you can. Year-over-year comparison confounds AI adoption with every other change in the same window — team composition, product phase, technical debt. A cohort comparison between teams with and without access is far cleaner, and is broadly what the stronger 2026 studies did.

Measure over a quarter, not a sprint. The gains appear immediately and the costs appear late. A two-week measurement window is structurally biased toward a positive result.

Tag provenance at merge so you can attribute rework and incidents later. This is a small amount of engineering work that makes every subsequent question answerable.

What this means for the rollout decision

None of this argues against adopting coding agents. Adoption is close to universal and the tools genuinely help, particularly for developers onboarding to unfamiliar code, for boilerplate, for test generation and for exploratory work.

It argues against justifying adoption with a single flattering metric, then being surprised when delivery velocity does not move the way the business case predicted.

The teams getting sustained value share a few habits. They invest in codebase conditions — types, tests, clear conventions — because those multiply agent effectiveness far more than tool choice does. They treat review capacity as the new constraint and staff for it. They instrument token spend per team from the start. And they keep measuring after the pilot ends, which is precisely when most measurement programmes stop.

If you are planning or reviewing a rollout, our AI coding agent rollout playbook covers the adoption mechanics, and this piece covers whether it worked.

Ortem Technologies runs AI-assisted delivery across client engagements and instruments the outcomes rather than assuming them. If you want help establishing baselines and measurement before a rollout, see our dedicated development teams, our staff augmentation practice, or talk to our team.