Mixed Signals on AI Impact in Software Development

METR, a non-profit research organization with a mission to evaluate the capabilities and risks of emerging AI systems, recently released a study that measures the impact of AI coding assistant tools such as Cursor and Copilot on the software development process. To put it mildly, the results were surprising, and stirred up a healthy amount of debate. Briefly, METR found that, at least within the scope of their study, AI coding tools actually slowed software developers down by 19%!

This result is in stark contrast to a number of other studies. For example, the widely read and well respected DORA State of DevOps Report for 2024 found that increasing adoption of AI coding tools by 25% improved software engineering productivity by 2% per capita. And a Jellyfish study (as well as previous versions), looking at actual development activity via data in systems like Jira and GitHub, found that AI is delivering positive impact (e.g., 16% faster PR cycle times on average), and that these gains are steadily increasing over time as tools and models get better. Per this same Jellyfish study, quality as indicated by the number of bugs issues introduced was not impacted by AI use.

On the surface, it seems hard to square the METR results with these other more encouraging stats.

To their credit, the METR researchers were very clear and specific about their study approach, and did an exhaustive job of describing the limits of what we can infer from their results based on that design.

As a brief overview, their approach was based on looking at the normal work of experienced developers working in open source projects. Focusing on a group of experienced (3+ years) open source developers, they compared productivity when using AI tools versus working without AI tools. Participants worked on normal issues in their respective projects, which were selected to focus on large, mature, high quality open source projects. Participants worked on a set of normal tasks over the period from February to June of 2025. And for each task, they were at random allowed or disallowed from using AI tools. When using AI tools, participants were allowed to select any tools of their choosing, although Cursor Pro with Claude 3.5/3.7 Sonnet was the most popular choice at the time of the study.

The standout result is clearly the impact seen on issue cycle time, where participants took on average 19% longer to complete issues. And perhaps equally surprising and a bit puzzling, when asked to self assess their speedup, participants reported expecting a 20% speedup even after experiencing the actual 19% slowdown! Participants persisted in believing that AI would speed up their work, even after it had observably slowed them down!

This obviously raises concerns about studies of AI impact based on self reported assessments of productivity. But of course, we still have the hard and fast results of studies based on system data to reconcile, so a mystery remains.

Of course, there are many possible causes for this result. And in fact, the METR paper is exceptionally thorough in outlining these possible causes. Some examples – certainly not an exhaustive list – include things like:

• Participants were not trained or highly experienced on the AI tools – it’s possible with more experience or enablement they could have achieved better results.

• Participants were intentionally selected to be highly experienced and ramped in their projects. Looking at a more diverse population of experience and tenure levels might show greater impact for AI tools.

• The open source projects in question were selected to be of the highest quality and maintainability. And generally, open source projects can tend towards very high levels of quality given the goal of supporting meaningful involvement across diverse and distributed groups of contributors. This extreme quality goal may magnify the effect of needing to re-work code suggestions from AI to conform to project standards.

These are just a few examples of very plausible root causes, and the study writeup is methodical and comprehensive in outlining the space of such hypotheses. Of course, it’s impossible to be sure which of these hypotheses actually holds without further experimentation, but intuitively, it seems like some of these concerns almost certainly had some effect on the results.

So what should we take away from these study results? Certainly there’s a very scientifically grounded fact that has been established here, which is that for some specific set of code repositories, and a specific set of AI tools, and a specific set of tasks, there are cases where we can expect AI tools to slow us down – at least given the versions of these tools at the time of the study (early 2025).

So what? Anyone working in software leadership could have told you that. We all have friends in the industry who rolled out Copilot to their team hoping for magic, only to find those first couple of months of velocity reports showed really concerning slowdowns.

What we want to know from the study is what it tells us more generally about the practice and future possibilities of AI assisted software development.

I take two main conclusions away from the study:

1. We have to get away from looking at point in time results, and look instead at trends over time.

2. We must not deploy or study AI coding tools in isolation, but rather as parts of more comprehensive programs. Don’t ask, “Is this AI tool improving productivity?” Instead ask, “Is my overall DevAI program improving productivity?

Just like the Eng leader looking at a monthly slowdown on their team, looking at the slowdown in the METR results tells me nothing about what is possible with DevAI, even for the very narrow scope of projects, issues, and developers considered. We need to start looking at results over time.

The METR study represents a bit of a torture test for AI tools – picking projects with the most stringent style and quality guidelines and trying to speed up the most experienced developers by giving them relatively unfamiliar tools is almost certainly not going to give immediate speedups. But I wouldn’t be at all surprised if those developers improve at using AI over time. Even within the METR study itself, which ran for five months, it would have been possible to look at results month by month, or slice it by cohorts based on how much time the participants put into working with AI tools. I bet that would show incremental improvements over time.

Would looking at the trend in this extreme case actually show a path to netting out positive gains on some timescale? We can’t be sure. But it’s also certain that the tools are improving, and knowledge of best practices in areas like prompting and leveraging useful MCP integrations is rapidly improving as well. These gains multiply against the experience level of the developers using the tools. I suspect that looking at the net trend here would hint at positive gains in the near future.

We also have to start looking at adopting AI tools in software development as a program of integrated actions – not just as deploying a tool. To be clear, it’s vitally important that we get more scientific about how we measure the impact of DevAI. Expectations are extremely high, anecdotes of amazing results abound, blogs and media about engineering have shifted almost all focus onto AI. With this kind of unprecedented wave of change in the industry, we absolutely need to get serious about measuring outcomes, and fast! But what are we measuring?

The variables in question absolutely include things like which tools we deploy, which models we leverage, which use cases we focus on, and which features of the tools (autocomplete, chat, agentic, etc.) we use. We need to study how those decisions influence productivity. But equally important is the need to measure other aspects of how AI is rolled out into dev teams. What training or enablement is provided? What mechanisms for team feedback and adjustment are established? What kinds of specialized roles or processes are introduced, such as designating AI experts to curate best practices for the organization, establishing guilds or channels for sharing, running AI demo sessions, lunch and learns, or hackathons to spread visibility to techniques and results.

These types of investments have a big impact on outcomes. Teams need to get good at working with AI. This includes techniques like learning to prompt well, and getting thoughtful around task granularity which can be very issue-dependent. Teams need AI-oriented documentation, such as “rules” for the AI coding tools that capture style and practice guidelines, or agent markdown files in each repo to give the tools context around design, repo-specifc patterns, non-functional objectives, quality and testing guidance, etc.

Developers and teams don’t wake up one magical day suddenly able to do an awesome job at all of that (and more), and in a way that makes contextual sense for their team and their org. It takes time and investment, all of which is supported by the types of programmatic investments above.

So if you really want to measure what matters, certainly ensure that you’re able to capture the metrics you care about independent of AI, and ensure that you’re trending those over time. And when you look at results, don’t just ask, “Is the tool working?” Ask, “Is our DevAI program doing enough to maximize our gains?”

Finally, it’s worth noting that, in terms of understanding the impact of AI in your organization, having comprehensive baseline metrics in place is essential. In a focused study like METR, they could zoom in on a single metric of talk time. After all, given the study participants being senior contributors well established in their respective open source communities, we could take it as a given that the work would meet other goals such as code quality.

For leaders rolling out AI in a more general engineering environment, we can’t make those kinds of assumptions. We absolutely want to look at velocity, but we also need to look across a variety of metrics that tell a fuller story of productivity – flow metrics, quality, etc.

Getting a good handle on baseline metrics, including the ability to trend these over time, and starting to include data around AI adoption, is the foundation to driving a solid AI program.

Studies like METR are invaluable for keeping the bar high around how we measure AI impact, and in providing a good reality check that just because there are lots of positive stories and bullish predictions about DevAI impact doesn’t mean you’ll magically get those gains quickly, easily, or in every case. But when I stack one study with negative results up against the volumes of positive evidence that I see in other studies and cases, it’s hard to ignore that we’re in the midst of a software engineering revolution that will have a very positive impact on the software that we as an industry can produce.

Let’s make sure we’re measuring the right things to get to that positive future as quickly and efficiently as possible.