Coding Agents in Production
What controlled trials, delivery data, and the benchmarks themselves say about AI-assisted software engineering
Executive summary
Engineering leaders are deciding how far to lean on coding agents on the strength of vendor demos and leaderboard scores. The published evidence is more mixed than either suggests: two controlled trials point in opposite directions, delivery telemetry shows stability falling as adoption rises, and the benchmark most often quoted has been retired by one of its own maintainers. This report sets out that evidence and the measurements a team should run on its own codebase before it changes how it works.
Key findings
- The two best-known controlled trials disagree. Developers finished a scoped, greenfield task 55.8% faster with Copilot in 2023; experienced maintainers finished real issues in their own repositories 19% slower with early-2025 tools, while believing they were 20% faster.
- At the organisation level, a 25% increase in AI adoption was associated with a 1.5% fall in delivery throughput and a 7.2% fall in delivery stability in the 2024 DORA survey, alongside gains in individual productivity and satisfaction.
- Code quality telemetry moved the same way. Across 211 million changed lines, refactored code fell from 25% of changes in 2021 to under 10% in 2024, and duplicated blocks rose from 8.3% to 12.3%.
- Developer trust has fallen as usage has risen. In 2025, 84% of developers used or planned to use AI tools, 46% distrusted their accuracy, and 66% said they spend more time fixing code that is almost right.
- The benchmark most often quoted is no longer fit for the purpose. OpenAI found flawed tests in 59.4% of the SWE-bench Verified problems its model failed, plus contamination, and stopped reporting it in February 2026.
Cite this report
Arcifa, R., & Carra, F. (2026). Coding Agents in Production. Montana Research Foundation report MRF-R-2026-02. https://montanaresearch.org/reports/mrf-r-2026-02/
BibTeX
@techreport{arcifa2026coding,
title = {Coding Agents in Production},
author = {Arcifa, Ricardo and Carra, Francieli},
institution = {Montana Research Foundation},
type = {Insight report},
number = {MRF-R-2026-02},
year = {2026},
month = {8},
url = {https://montanaresearch.org/reports/mrf-r-2026-02/},
note = {PDF: https://montanaresearch.org/reports/mrf-r-2026-02.pdf}
}