Everyone who uses an AI coding assistant says the same thing: "I'm so much faster now." But what would we see if we measured that feeling with a stopwatch? In July 2025 someone did exactly that, and the result wasn't what anyone expected.
The experiment: 16 developers, 246 real tasks
In the first half of 2025, METR, an independent research organisation that measures the capabilities of AI models, ran a randomised controlled trial. Instead of an artificial task in a lab, they chose a setup as close to the real world as possible:
The developers had been contributing to these projects for years and had tens, even hundreds, of hours of experience with large language models. So there were no beginners and no unfamiliar codebases in the experiment.
The result: expectation, perception and reality
When developers used AI, they took 19% longer to finish their tasks. The truly striking part is the second row: even after the experiment was over, the same developers believed AI had made them 20% faster.
So they didn't just slow down; they didn't notice they had slowed down. This is the productivity counterpart of the "overconfidence" finding from the Stanford study I mentioned in the first post.
Why did they slow down?
METR examined 20 possible factors that could explain the slowdown and reported that five of them likely contributed. The setup of the experiment also hints at why the result may not be that surprising:
- A familiar house, a new guide. The developers knew their codebases by heart. Where they were already fast on their own, explaining the context to the model was an extra cost.
- A high quality bar. In a project of a million lines used by thousands of people, "it works" isn't enough; expectations for tests, style and design are high. Bringing the model's suggestion up to that bar takes time.
- Waiting and reviewing. Working with AI can shorten writing time but lengthen reading, waiting and fixing time. What you feel is writing speed; what gets measured is total time.
A year later: METR's February 2026 update
The story doesn't end there. In February 2026, METR asked the same question again with late-2025 tools, meaning agent-based tools such as Claude Code and Codex. This time there were 57 developers, 143 projects and more than 800 tasks.
For the developers from the first study, the direction changed: from slowing down to speeding up. But METR itself presents this result with caution, because the confidence intervals are very wide and include zero. More interesting still, the organisation's main concern runs the other way: they think the real speed-up may be larger than what was measured.
Here's why: the developers most used to AI no longer accept doing half their work without it. Between 30% and 50% of participants said they never submitted tasks they didn't want to do without AI. And measuring the time of people running several agents at once became almost impossible. So METR concluded that the measurement didn't reflect the real effect and announced it was redesigning the study.
Signals from the field
Alongside the controlled experiments, large surveys show a similar tension: usage is rising fast, but trust isn't.
DORA's finding is especially instructive. As AI adoption rises at the team level, individuals feel more productive and more focused; but that doesn't carry over to software being delivered quickly and safely. The possible explanation the report puts forward: when code is written faster, changes get bigger, big changes are harder to review, and the risk of errors goes up.
So what should we do?
The conclusion from this data isn't "stop using AI". METR itself says clearly that its study doesn't show AI is useless in software engineering; it even notes that AI may help less experienced developers, or those working in unfamiliar codebases, more. The conclusion is simpler: a feeling is not a measurement.
- Understanding an unfamiliar codebase or library
- Boilerplate, test scaffolding, data transformation
- Quick prototypes and throwaway scripts
- Interpreting an error message, producing a first guess
- A high-standard codebase you know by heart
- Work where implicit rules and history matter
- Changes where security and data integrity are critical
- The review cost of "almost right" output
Three suggestions for a team:
- Run your own measurement. For a few weeks, record time, number of bugs and review rounds on similar tasks. Compare the speed you feel with the speed you measure.
- Measure delivery, not writing. How long a feature takes to reach production, and how many bugs it causes there, is a far more honest indicator than lines of code written.
- Make review cheaper. Small changes, automated tests and security scans in CI are how you use AI's speed safely. The 19 checks in the vibe coding post exist exactly for this.
AI tools change every six months, and some of the numbers in this post will soon be out of date too. But the lasting lesson from METR's first experiment is this: we are poor witnesses of our own productivity. The stopwatch is always more honest.
Sources
- J. Becker, N. Rush, B. Barnes, D. Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, July 2025.
- METR, Uplift update: late-2025 AI tools, February 2026.
- E. Paradis et al., How much does AI impact development speed? An enterprise-based randomized controlled trial, Google, ICSE 2025.
- S. Peng, E. Kalliamvakou, P. Cihon, M. Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, 2023.
- Stack Overflow, 2025 Developer Survey: developers remain willing but reluctant to use AI, July 2025.
- InfoQ, 2024 DORA Report, November 2024.