In July 2025, METR ran the study everyone had been asking for. Sixteen experienced open-source developers, 246 real issues, repositories they already maintained and knew well. Half the issues allowed AI tools, half did not. The developers forecast they would go 24% faster with AI. They went 19% slower.
The number travelled. It appeared in newsletters, in conference talks, in the comment sections of every productivity debate, usually stripped of the sixteen-developer sample and the open-source caveat and reduced to a single line: AI makes developers slower.
On 24 February 2026, METR published a follow-up. Fifty-seven developers this time, 143 repositories, more than 800 tasks. The new estimate was an 18% speedup for returning developers, with a confidence interval running from −38% to +9%, and a 4% speedup for newly recruited developers, interval −15% to +9%.
Both intervals cross zero. METR's own reading of its own data: the new experiment "gives us an unreliable signal of the current productivity effect of AI tools", and its central estimate is "likely a bad proxy for the real productivity impact."
That is about as close to a withdrawal as a research organisation gets in public. Nineteen months after the first result, the people who produced it are telling you not to lean on the successor, and by extension not to lean on the headline it replaced.
The citations have not noticed.
Why the second study failed, and why it matters more than the first
METR is specific about what went wrong, and the reason is worth more than either number.
Developers would not do the work. METR reports that 30% to 50% of participants said they were choosing not to submit some tasks because they did not want to do them without AI. The control condition had become unattractive enough that the sample selected itself.
Read that twice. The experiment could not hold its control group because working without the tool is now something experienced developers actively avoid. That is not a measurement of productivity. It is a measurement of dependence, and it arrived as a methodological failure rather than as a finding.
It also means the honest answer to "does AI make developers faster" is currently we have lost the ability to run the experiment cleanly. You cannot randomise people into a condition they will quietly refuse.
The number on the other side is no sturdier
If METR is the headline for the sceptics, the 55.8% figure is the headline for everyone else, and it deserves the same treatment.
It comes from a 2023 randomised trial in which developers with GitHub Copilot implemented an HTTP server in JavaScript 55.8% faster than a control group. The confidence interval runs from 21% to 89% — a range wide enough to contain two completely different conclusions. The sample was 95 freelancers recruited on Upwork. The task was one synthetic greenfield exercise with no existing codebase, no code review and no maintenance. Three of the four authors were at Microsoft or GitHub.
None of that makes it wrong. It makes it a measurement of something narrow: how quickly a freelancer completes a self-contained exercise. It is not a measurement of shipping software.
The strongest evidence on the positive side is quieter and almost never quoted. Three randomised trials at Microsoft, Accenture and a Fortune 100 manufacturer, pooled across 4,867 developers in real jobs on real repositories, found a 26.08% increase in completed tasks. The authors' own summary of their primary outcomes is that "only the effect on the number of pull requests is statistically significant at conventional significance levels." One Accenture arm of the same study produced a point estimate of −39.18% on completed pull requests, which sits in an appendix after a layoff removed 42% of the participants.
So the best pro-AI field evidence is a modest, partly insignificant positive, with one arm pointing the other way. The best sceptical evidence has been disowned by its authors. Anyone telling you the question is settled is citing one of these four numbers without its interval.
What survived
One finding from the 2025 METR study has survived every revision, and it is the one nobody repeats.
The developers forecast a 24% speedup. They measured 19% slower. Afterwards — having done the work, having lived through it — they still estimated they had been 20% faster. The gap between what they experienced and what the stopwatch said is about 39 percentage points, and it did not move when the effect size did.
METR's later survey makes the same point from the other direction. Asked directly, 349 technical workers reported a median 3x speed increase from AI. METR publishes that figure alongside its own warning that "survey results are not necessarily grounded in reality", and notes that its earlier participants overestimated the effect by roughly 40 points.
So the durable, replicated result of this entire literature is not a productivity number. It is that practitioners cannot estimate their own productivity with AI, and that the error runs in one direction: everyone thinks they are faster than the measurement says.
That is a more actionable finding than either headline. It means self-reported adoption data — which is most of what exists, including the 2025 DORA survey where more than 80% of respondents believed AI increased their productivity — is measuring belief. It is real data about what people think. It is not data about output.
The quality question is separate and better evidenced
Lost in the argument about speed is that the evidence on what the code is like is more consistent than the evidence on how fast it arrives.
DORA's 2025 report finds AI adoption positively related to delivery throughput and still negatively related to delivery stability, and lands on a formulation worth keeping: "AI doesn't fix a team; it amplifies what's already there." Veracode, testing more than 100 models against 80 coding tasks, found 45% of samples failed security tests, and reported that the models had got better at writing code that works without getting better at writing code that is safe. Two thirds of developers in Stack Overflow's 2025 survey named "almost right, but not quite" as their single biggest frustration with AI, and 45.2% said debugging AI-generated code takes longer.
Both DORA and Veracode sell into this market, which is worth stating. It is also worth noting which direction their incentives point: a vendor reporting that its category does not improve security is not telling you what it would prefer to tell you.
What we take from it
We back companies whose teams are small and whose engineers are early in their careers. The question of whether these tools make a junior developer effective is not academic for us; it decides what a seed-stage team can attempt.
The honest position today is that the measured answer is unavailable, the self-reported answer is inflated by about 40 points, and the one result with any consistency across studies is that output quality tracks the team rather than the tool. The three-company trial does hint that gains concentrate in less experienced developers, 27% to 39% against 8% to 13% — and its authors say plainly those estimates are noisy and not significant. We are not going to cite them as if they were.
What we do with that is simple enough. We stop asking founders how much faster AI makes them, because the literature says they cannot know. We ask what shipped, to whom, and whether it held.
None of this argues against the tools. It argues against a specific move that has become common in pitch decks and policy papers alike: taking a measured effect from one study, stripping its interval, and treating it as a constant. The most-cited number in this debate was withdrawn by the people who measured it, and it is still in circulation. That is the finding.