The Metrics Trap: when measurement becomes the mission

Eons ago I spent a lot of time helping organisations develop information management and reporting systems. I got loads of exposure to management meetings and papers, heaps of data and charts and how management decision-making utilises different metrics. It convinced me of the importance of accurate, trust-worthy data.

I then got management responsibility myself with all the associated the metrics and decision making. My scope of responsibility was software product delivery, with multiple teams building and releasing changes, lots of customers interacting with systems and websites and a hundred and one things happening every day. I used data to keep across what was happening, identify improvement opportunities and demonstrate the value that my teams were bringing to the organisation.

But I also saw challenges with metrics. There are so many things that you can measure - a manager can build a cottage industry to produce and review reports and dashboards, or worse produce and ignore.

Instrumentation and Automation enables almost endless production and processing of data. I just asked perplexity how much data is produced each day and here’s some of the response:

Every day, approximately 402.74 million terabytes of data are generated globally. This figure encompasses all forms of data that are newly created, captured, copied, or consumed (1, 2, 9).

To put this into perspective, this daily data creation translates to about 147 zettabytes annually, with projections indicating that this number could rise to 181 zettabytes by 2025 (1, 2). The exponential growth in data generation is driven by various factors, including increased internet usage, the proliferation of connected devices, and the growing reliance on digital communication platforms.

Breakdown of Daily Data Generation

• Emails: Approximately 361 billion emails are sent daily8.

• Social Media: Platforms like Twitter see around 500 million tweets sent each day (8).

• Search Queries: There are about 5 billion searches conducted on search engines daily (8).

• Video Content: Videos account for over half of internet data traffic, significantly contributing to the total data generated (1).

The metrics trap starts with the opportunity to spend all your time processing data in search of insights. The endless possibility of what you can measure, means you need to be selective about what you actually measure and pay attention to.

So where to start? Industry “best” practice is not an unreasonable place to start - after all, if it’s worked for other people, chances are it can work for you, right?

I wrote about using DORA metrics to measure software delivery, which are definitely in the category of industry best practice, and are a good place to start.

I have been watching “the discourse” over developer productivity. This has always been a debate and a focus, despite plenty of warning that, whilst historically expensive from a direct cost perspective, local optimisation of typing code is a case of failing to observe the wood because of all these damned trees.

And there’s something worse. Only focusing on quantifiable outcomes can lead us down a dangerous path – one where we risk optimising for what we can measure rather than what truly matters.

Consider a software development team tracking story points as their primary measure of productivity. Initially, this metric provides useful insights into team velocity and capacity planning. However, as management begins using these points for performance evaluation and team comparisons, something else happens: Teams start breaking down complex stories into smaller ones, inflating point estimates, or choosing technically simpler solutions to boost their numbers.

Goodhart's Law tells us that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." In other words, the moment we turn a measure into a target, it ceases to be a good measure. People will find a way to game the system.

In researching more about measurement and management I came across the McNamara fallacy, named after Robert McNamara, US Secretary of Defense during the Vietnam War. McNamara became fixated on quantifiable measures, ignoring qualitative data.

The progression is familiar in modern organisations, albeit, hopefully, not with such devastating impact:

1. We begin by measuring what's easily quantifiable

2. We dismiss what can't be easily measured

3. We gradually presume that unmeasurable factors aren't important

4. Finally, we convince ourselves that what can't be measured doesn't exist

It is far too easy to become fixated on velocity metrics while overlooking crucial factors like team morale, knowledge sharing, and sustainable technical practices – elements that impact long-term success but resist easy quantification.

I’ve spent a lot of time travelling, and have experienced airports where ground crew are measured on the time taken for first bag to arrive on the baggage carousel. So what happens? One bag gets grabbed from the aircraft and run to the carousel. The time to the next bag? Who cares, other than the passengers waiting to continue their journey?

Back to software delivery. Consider these common developer efficiency metrics:

1. Lines of Code (LOC) - Perhaps the most notorious example of metric fixation in software development. Teams measured on LOC often produce unnecessarily verbose code, sacrificing maintainability for the appearance of productivity. One engineering manager shared with me how their team, under pressure to show "improved productivity," began avoiding useful third-party libraries in favor of writing everything from scratch – a classic case of Goodhart's Law in action.

2. Number of Pull Requests Merged - When teams are evaluated on PR counts, we can see a tendency to break changes into smaller, sometimes artificially separated commits. While smaller PRs should absolutely be encouraged, optimising for this metric could lead to fragmented changes that make it harder to understand the system's evolution.

3. Time to Close Tickets - Organisations fixating on ticket closure speeds frequently discover their teams breaking down complex problems into multiple smaller tickets or, worse, rushing through important technical decisions to maintain "velocity."

What these metrics fail to capture is often more crucial than what they measure. Just some examples in software development:

1. Technical Debt Accumulation - The most efficient code written today might become tomorrow's maintenance nightmare. Yet, how do we measure the quality of architectural decisions or the long-term maintainability of a codebase? These crucial aspects resist simple quantification.

2. Knowledge Sharing and Team Growth - A senior developer spending time mentoring others might show reduced "personal productivity" metrics while dramatically improving team capability. The impact of such activities often becomes visible only months or years later.

3. Innovation and Experimentation - Teams under strict efficiency metrics tend to stick to known solutions rather than exploring potentially better alternatives. The cost of this missed innovation is impossible to measure but profoundly real.

There's an old parable about a drunk man searching for his keys under a streetlight. When asked if he lost them there, he replies, "No, but this is where the light is."

This "streetlight effect" perfectly captures how organisations often limit their focus to easily measurable metrics while ignoring critical factors in the shadows.

Consider employee engagement surveys. While they provide some insight, they can't capture the nuanced dynamics of team collaboration, the impact of informal mentorship, or the innovation potential within your culture. Yet, these hard to measure elements often determine the difference between thriving and struggling organisations.

When I were a lad, Balanced Scorecards were a popular tool for businesses to look at how their business was tracking. Developed in 1992 by Kaplan and Norton, the intention was to look at value creation from four different perspectives:

• Financial perspective: Do your plans and processes lead to desired levels of economic value creation? Metrics include sales revenue, operating expenses, net income, and investment in assets.

• Customer perspective: Does your target audience perceive your product, services, and brand in the desired way? Metrics include quality, delivery speed, and customer service experience.

• Internal business process perspective: Do your organisational processes create value for customers? Metrics to track are related to operations and customer management, innovation, regulatory, and social processes.

• Learning and growth perspective: Does your organisation support and utilise human capital and infrastructure resources to meet goals? Areas to consider are human capital (people, talent, and knowledge), information capital (databases, networks, and technology), and organisational capital (leadership capabilities and cultural alignment to company goals), each with its own set of metrics.

Depending on your role within an organisation these responsibility for these perspectives may be way outside your wheel house. However, there is nothing preventing you from considering each angle with respect to your responsibilities.

I’m not specifically advocating for the use of Balanced Scorecards, rather the philosophy behind it. You absolutely need to use data to inform your decision making - Peter Drucker was not wrong that if you aren’t measuring something then your judgement about any changes is effectively no more than a guess. Data and measurement is absolutely necessary, but it is not sufficient.

To avoid the metrics trap requires you to approach measurement with awareness and nuance:

1. Embrace Qualitative Insights - Supplement quantitative metrics with rich qualitative feedback. Customer stories, team retrospectives, and direct observations often reveal what numbers alone cannot.

2. Question Your Metrics Regularly - examine whether your measurements still serve their intended purpose or if they've become targets that distort behavior.

3. Maintain Peripheral Vision - Actively look beyond your primary metrics. What important factors might you be missing? What unintended consequences could your measurements be creating?

4. Value Professional Judgment - Create space for experienced practitioners to make decisions based on wisdom and context, not just data points.

5. Look at Trends and Long-Term Outcomes - Typically software development is a team game contributing to overall business outcomes. What impact is the team having on customer satisfaction, either by delivering quality features, addressing persistent pain points, or, hopefully not, causing outages through poor execution? How is that changing over time?

Wherever I go, the businesses and teams that are the most productive and engaged and the ones that have a mature approach to using data in their decision making. I hope that everyone can reach that maturity.