A non-profit that evaluates AI models, METR, had high hopes. In July 2025, it published a controlled trial with 16 experienced developers and 246 real tasks in repositories they had known for years. The developers expected to be 24% faster and, once finished, with the measured times in front of them, they were still convinced they had gained 20%. But to finish the job, with AI assistants, they took 19% longer. The tools of that time already look dated, so METR repeated the experiment in the second half of 2025, with the tools of that moment, and published the result in February 2026: 57 developers, 143 repositories, more than 800 tasks. This time the veterans of the first study saved 18% of their time and the newcomers 4%, with error margins so wide that both results include zero, and with a detail the authors themselves highlight: between 30% and 50% of participants admitted setting aside tasks they did not want to do without AI. METR concludes that its figure is probably a poor indicator of the real effect and is likely an underestimate.
Today’s tools are better than those in that second study. What has not changed between one and the other is the gap between what gets measured and what gets felt. The measured improvement ranges from 4% to 18%, and the feeling, inside and outside the industry, is of a change of era. That gap is the same one we find in four sentences we hear almost every week, usually between the main course and coffee, and always with affection:
- “Now that you use AI, you must be raking it in”
- “Why is this so expensive if you do it with AI?”
- “Now you’ll be able to deliver faster”
- “With AI you’ll need fewer people”
All four are reasonable. All four start from the same idea: that AI makes what you buy cheaper. But it turns out, surprise surprise, that what you buy from an agency has never been about typing.
What you buy from an agency has never been about typing.
1. “You must be raking it in”
The first thing that changed was the bill, and not in the direction the sentence suggests. The visible line is the subscriptions to the coding assistants we use (Claude Code, Codex and a few more): several hundred euros a month. Below that line sit API usage of other models we need to test or use for specific tasks, hosting for the models we run in-house, hosting in general, cybersecurity, licences for management platforms and the specialists who keep all of that running. And we haven’t yet reached the truly expensive part: adapting the house so those tools actually work. Context files per repository, shared skills that capture how we do things, pipelines that review what the model proposes before a person sees it, and a rule that sounds simple and isn’t: when something fails twice, it gets investigated and fixed so it doesn’t fail a third time. That is hours from the most expensive people on the team, and it repeats every few months because the tools change name, price and capabilities at a pace no annual plan can keep up with.
Bart Butler, Proton’s CTO, calls it a collective action problem in an internal document: using AI well is something an engineer can do on their own; reorganising how an entire company works requires organisation and resources the company simply has not invested yet. We haven’t found it free either. Raking it in, not for now. What has changed is where the money goes.
2. “Why is it so expensive if you do it with AI?”
Because the code is the small part of your final cost. You pay for someone to decide what to build, what not to, in what order, with what content structure, and what happens when communications, IT and marketing want three different things. You pay for someone to read the whole tender, understand your business and say no to two ideas you liked. You pay for the result to genuinely meet AA accessibility, load fast and be maintainable by another team three years from now. And you pay for a named person to sign off every delivery. A model proposes, but it doesn’t answer for anything. Butler writes it in that same document: the engineer is accountable to themselves, to their colleagues and to the company in a way a model can never be.
That said, some work has become cheaper, and it’s only fair to name it: laying out documents, handling communications, tracking tasks, preparing proposals, research, documenting modules, managing maintenance. More or less routine tasks that rely on a base of information and context that only people could handle before. Look at the list: it’s our own work of running a project, and almost none of it appears as a line in your budget. That’s why it doesn’t disappear from it either. Those savings have gone into absorbing the new bill from the previous section and into giving time back to the part you do buy, which has also grown: reviewing more proposals, testing more routes and keeping the tools tuned is work that didn’t exist before. The day a task that does appear in your budget gets cheaper, you’ll see it on its line. What doesn’t go down is the part that was never mechanical, which in a complex web project is most of the price.
3. “Now you’ll be able to deliver faster”
Producing, yes. A module, a template or a test suite are ready sooner. What doesn’t speed up is the project, because a project is not a queue of coding tasks. Butler describes it precisely: when you remove one bottleneck, another takes its place. Perhaps continuous integration, maybe specifications, approvals or code review. The DORA 2025 report, Google’s annual study on software team performance, reaches the same conclusion by another route and describes AI as an amplifier that magnifies the strengths and weaknesses the organisation already had.
In a web project the bottlenecks have been in the same place for years, and none of them is the code: content that arrives late, sign-offs waiting for a meeting, decisions nobody wants to make until they see a screen. AI hasn’t changed a single one. They’re still there. And to be fair to the other side of the table: half the time the slow ones are us, because generating ten versions of something takes a minute and reading them with judgement takes an afternoon. Where you do notice the difference is elsewhere: within the same timeframe, we explore more alternatives before deciding, and fewer errors reach your inbox. More quality in the same time, which is not the same as the same thing in less time.
METR's two controlled trials with experienced developers, with and without AI assistants.
19 %
more time it took them to complete tasks with early-2025 tools
METR, 2025
20 %
of time they believed they had saved at the end of that same study, with the data saying otherwise
METR, 2025
18 %
of time saved by the same developers in the late-2025 rerun, published in 2026; the error margin includes zero
METR, 2026
4 %
of time saved by the developers who joined the second study; the error margin includes zero
METR, 2026
4. “With AI you’ll need fewer people”
We are 13 people. Our developers are still programming; what has changed is who they do it with. Code, Codex and their relatives work as a digital alter ego with an enormous capacity to understand a challenge, propose paths, discuss an architecture and, more and more often, write a solution that someone then reviews, corrects and signs off. Each person manages more capacity than before: more projects in parallel, more routes explored in each one and more time on the part of the job that consists of talking to you. The team isn’t shrinking; what each person does is changing.
What happens when someone believes the equation
The Butler document we have quoted twice is titled “AI @ Proton, Engineering Manifesto”. 404 Media published it on 29 October 2026 as part of an investigation into how Proton, a company that sells privacy, is pushing AI internally. The manifesto’s diagnosis largely matches this article’s: expertise isn’t dead, good practice isn’t either, the bottleneck shifts, and reorganising a company requires investment. What came next is what happens when leadership keeps only the first sentence: that programming is dead.
According to 404 Media, in late August an employee asked in an internal security channel for a few days to run penetration tests on new code before releasing it. The CEO, Andy Yen, replied that this was “bullshit and loser thinking”, asked people to stop crafting code by hand so lovingly and to use more AI, and announced that this would be the last year Proton hires engineers. The company installed software on computers to measure how much code AI generates and planned to re-interview staff about their use of these tools. An internal document cited by the outlet says Proton will miss its recurring revenue target because of insufficient speed. Proton denied none of it. In fact, its spokesperson acknowledged that the company had been too blunt, or too slow, in explaining itself.
Read it slowly: more AI, fewer people, fewer tests, measuring generated lines instead of results, and the revenue target missed. It’s those four recurring sentences people keep throwing at us, applied to a real company. We would sign almost all of its CTO’s diagnosis; what the case shows is that speed without judgement only produces speed.
What to ask any supplier that “uses AI”
If you’re worried that the price doesn’t reflect the tools, these questions will serve you better than asking for a discount.
Five questions before you sign
The answers say more about the supplier than any presentation.
Ask them to name specific tasks that cost less than two years ago and how that shows in the budget. If the answer is "everything" or "nothing", be equally wary.
One named person for every delivery. "The team validates it" is not an answer.
Results in production (errors, accessibility, performance, deadlines met) or volume of generated code. The latter is what Proton measured.
How it's detected, who fixes it and whether that's included in the price or billed separately.
If the answer is "nothing", they have a subscription, not a method.
Sources
- METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 2025 — the controlled trial with 16 developers and 246 tasks: 19% slower with AI, and the gap between what was expected and what was measured.
- METR, We are Changing our Developer Productivity Experiment Design, 2026 — the rerun with 57 developers and more than 800 tasks: savings of 18% and 4% with intervals that include zero, and the authors’ warning about the reliability of the figure.
- Joseph Cox, 404 Media, Manifestos, Fear, and ‘Loser Thinking’: Inside Proton’s AI Reckoning, 2026 — the investigation that published the manifesto and documents Andy Yen’s statements, the measurement software and the missed revenue target.
- Bart Butler, AI @ Proton, Engineering Manifesto, internal Proton document published by 404 Media, 2026 — the quotes on the collective action problem, the engineer’s accountability and the shifting of bottlenecks.
- DORA (Google Cloud), State of AI-assisted Software Development, 2025 — AI as an amplifier of the strengths and weaknesses already present in the organisation.