Unlocking Data Hygiene for Engineering Metrics

Industry attention on the topic of measuring engineering productivity has increased noticeably in the past couple of years. The rapid transformation of software development with AI is bringing a new level of urgency and focus onto this problem. As we quickly adopt new AI tools and change our development processes, we want clear quantitative feedback about resultant changes to productivity, including factors like volume and velocity of output, software quality, and overall costs of development and maintenance. How can we possibly answer these questions without a solid engineering metrics strategy?

I’m betting that AI assisted software development will land in a positive place, with AI augmenting human teams to produce greater value and innovation faster and at lower cost. But wherever the “DevAI” space leads, I suspect that it will have a positive impact on the practice of engineering measurement. The space program may not have actually created Tang, but it did unlock dozens of inventions we use every day, from camera phones to memory foam. I expect the AI revolution will have similar benefits for engineering metrics.

One of the most common concerns I hear from engineering leaders around introducing or renovating their metrics strategies is around the underlying data required. Especially in orgs that are scaled and have a bit of history, I hear data concerns such as teams in the org using different tools and processes, teams being inconsistent about what they track in systems and how they track it, and more. “Our data is a mess,” is a common refrain and often cited blocker for advancing a metrics strategy.

But what does it mean for the data to not be “a mess?” I like to call this property of engineering environments “organizational observability.” We care about our software systems being observable so that we can monitor their operations and diagnose issues when they arise. It only makes sense that we should care equally, if not more, about our workflows and development processes also being “observable.”

A key insight I’ve had over time is that the organization doesn’t need to be perfect in order to be observable. Nothing like it! We just need to achieve the level of process and data consistency that enables us to measure the metrics we care about reliably and consistently. Focusing on the actual requirements of organizational observability instead of some broader notion of process and data perfection is a critical unlock, allowing a pragmatic approach that can deliver quick value.

Organizational observability is something we can approach incrementally, and we don’t need data perfection or absolute process consistency. But we do need to get the essential data into a good enough place to support needed metrics. The alternative is managing by feeling and word of mouth, which just doesn’t scale.

But how do you proceed? The idea of organizational observability can seem daunting, but in reality it’s very approachable. And the ROI of investments here tend to be well worth the effort.

Organizational observability means having the data in place to accurately compute the metrics that the organization selects to measure engineering productivity and performance. This may seem a little circular – needing a handle on your metrics strategy to define organizational observability requirements – but it works in practice.

For one thing, better standards for defining a top level metrics model have been emerging in recent years, such as the DORA and SPACE frameworks, and the Github ESSP model. So establishing an initial cut at a metrics model is a lot easier than it was a decade ago. Even in cases where the business context dictates using more custom metrics than suggested in the standards, the standards still provide a framework for thinking about best practices such as considering the broader set of software development concerns that are in tension and often pose trade-offs, such as quality versus velocity.

Starting with our metrics model, we can focus on implementing the truly necessary set of process and data practices needed for measurability, rather than attempting broader and riskier changes. Small focused changes prosecuted vigorously maximize our chance of success.

To make this more concrete, let’s consider a realistic example. I find that the Github ESSP provides a solid, general purpose starting point for considering a metrics strategy. And I like to add in a couple of bonus metrics that seem to be consistently overlooked in the standard models, but end up being important in many real world environments. For example, I’ve added a predictability measure that I’ve never been able to skip in most exec and board reporting cases.

An initial simplified metrics model starting with these ingredients might have the following elements:

• Business Outcomes / Allocation - Percent of engineering investment allocated to feature development versus maintenance and reactive work.

• Velocity / Epic or Ticket Cycle Time - Average time from work being picked up to code being shipped.

• Velocity / Deployment Frequency - Deployments per unit time (e.g., hourly, daily, weekly etc.)

• Quality / Change Failure Rate - Percent of deployments that cause a production failure.

• Quality / Defect Rate - Percent of engineering effort spent on fixing bugs.

• Developer Happiness / Developer Satisfaction Score - e.g., average percentage of positive responses across survey questions about the engineering environment.

• Bonus / Predictability - Percentage of initiatives delivered on time and on scope.

• Bonus / Customer Value - Highly context dependent based on the product and business model, but common approaches look at adoption measures such as daily active users.

This is a simplified model. For example, in each of the categories we’ve only selected one or two metrics, while the ESSP includes three per category. But this isn’t so simplified as to not be realistic.

Starting with these sample metrics as our initial model, we can readily determine a high level picture of our data and process requirements. What do we actually need from the org to drive our “version one” metrics strategy and organizational observability?

Considering this question metric by metric:

• Business Outcomes / Allocation - Being able to determine our actual allocation of effort and investment requires that all work be ticketed (i.e., that we have an easy way to “see” all of the work). Ideally work should be organized into epics (or some other macro, initiative level ticket), which makes it easier to inspect abnd validate the accuracy of our data, and easier to add and manage dimensions of interest. For example, if my top level allocation buckets include things like Roadmap, Customer Driven Work, Defects, and Maintenance, then we need a clear and reliable way to tag and bucket tickets and initiatives into these categories. This level of structure in our ticketing system is not a huge imposition, and is often jointly supported but the product org.

• Velocity / Epic or Ticket Cycle Time - Flow metrics such as epic or ticket cycle time are critical indicators of engineering productivity. Issue cycle time requires that tickets have clearly identifiable lifecycle events such as work being started and work being completed. It’s very common for teams to adopt their own custom workflows in ticketing systems like Jira, but these custom workflows typically still provide ways to indicate macro lifecycle events such as the ticket being closed, even if the close state varies, e.g, closed/done, close/won’t fix, etc. Again, we see that the actual need here is not a huge imposition or lift. Teams can keep their custom workflows. We don’t all need to adopt “one way to use Jira” (as if!). We just need some common definitions about lifecyle.

• Velocity / Deployment Frequency - In service oriented environments, measuring the frequency of deployment is a key productivity indicator. Being able to ship more frequently enables smaller units of work and better, faster workflow. Measuring this requires instrumenting CI/CD pipeline to determine when deployments happen. Again, teams will often have varying CI/CD setups, but as long as they can raise an event or record an update in a consistent manner on deployments, we have all we need for this measure.

• Quality / Change Failure Rate - Similar to deployment, here we need our CI/CD instrumentation to record success versus failure on deployments, and ideally we also want our production monitoring system to be able to raise an event or record an update on any production failure. This information is almost always readily available, it just needs to be captured into a central repository.

• Quality / Defect Rate - Measuring the amount of effort (e.g., number of issues, time spent on issues, etc.) related to defects provides a good indicator of quality concerns. Similar to allocation, this measure requires that we have sufficient tagging on tickets, or are using readily identifiable issues types such as “Bug,” to reliably identify work on defects. Again, this is not typically a big change, if any, to how most teams are working. While consistency across teams makes life easier, it isn’t strictly necessary as long as the rules for identifying defect tickets are clear on a per team basis.

• Developer Happiness / Developer Satisfaction Score - This requires that we have a system to run developer surveys, which is an increasingly universal best practice. There are lightweight starting points like Google Forms that are serviceable before investing in more sophisticated survey tools.

• Bonus / Predictability - Determining predictability is about being able to hit delivery commitments, mainly around timelines and scope. While determining scope is a qualitative judgement, timelines can be objectively measured. This requires that initiatives (e.g., represented by Epics) include target dates, at whatever granularity makes sense for the org.

• Bonus / Customer Value - Measuring customer value is very product and business specific. Basics like measuring active users are typically readily available via systems like Pendo. But for many products, you want deeper and more specific value measures, which should be jointly designed with the product organization to align with business outcomes. For example, a workflow application may want to measure things like tasks completed per-capita, or a collaboration application may want to measure projects and comments. Whatever the value proxy, it’s healthy to build a culture of ensuring observability to product business value.

So now we know what we need to achieve in our data. But how do we get there? Of course, the standard tactics of change management apply here, including broadly communicating a clear strategy and plan, ensuring strong leadership is in place for the work, involving the team and soliciting feedback, and celebrating wins and value creation along the way. But the particular challenges of organizational observability inform some more specific helpful tactics.

As with any organizational change, introducing or updating a metrics strategy requires thoughtful design on the basis of clear goals. But in my experience, engineering metrics require extra effort on this front. Introducing a bunch of bookkeeping into software development work, which is already often difficult and stressful to begin with, especially if the benefits are some vague notion that upper management can make sure we’re working hard enough, is just a DOA premise. The design needs to be thoughtful and non-disruptive. And above all, the goals and intended benefits need to be clear and positive.

The good news is that, on a fundamental level, engineering metrics can and must serve a positive purpose for the whole organization and everyone in it, not just upper management. Metrics may be introduced to measure specific initiatives, such as improving product output by scaling the team, making more time for innovation by first surging to reduce technical debt, etc., But for the most part, the overarching goal of engineering metrics is to make sure that the engineering organization is driving the product strategy of the company as efficiently as possible, blending factors such as capital efficiency, velocity, agility (responsiveness), and deep innovation as appropriate to the company's overall strategy.

Put another way, engineering metrics are the telemetry to let us know we’re winning, and to help us diagnose any areas where we could be doing better. This is in the direct vested interest of everyone on the team.

But communicated carelessly, it may not seem that way. And if metrics are cast as just overhead for management, or worse yet, a way to keep tabs on the team and push for more output, it’s hard to get much buy in on the effort to make the organization observable, which often falls to individuals following the necessary process steps and ensuring the right data is entered along the way.

Sell the value of metrics and at least you’ve bought yourself an initial burst of momentum.

Assuming the goals of the metrics strategy are thoughtfully designed and communicated, the next step is showing clear value from metrics. Most software teams are placed under great pressure, typically building new stuff while maintaining what’s already been shipped, all while fielding support issues, helping with roadmap planning, and more. If updating a bunch of data to power metrics doesn’t provide tangible value, it won’t happen. And with less attention, data can quickly degrade into a lower quality state where it can then be harder to then show its value – a negative reinforcement loop!

The solution is showing quick value, even if it’s basic. A useful metrics view, at least one aligned with the goals described above of helping the team succeed, is diagnostic in nature. That is, our metrics shouldn’t just measure output, but should look at multiple factors that give a picture of overall team function.

Design for this diagnostic purpose is necessarily balanced and multi-dimensional. It’s often noted about good engineering metrics design that a metrics strategy should be careful to account for Goodhart’s Law, which states that when an individual measure becomes a target, it ceases to be a good measure, as simple single-dimensional measures are often easily gamed.

The solution to avoiding the Goodhart trap is simple: ensure that your metrics strategy is well balanced and considers a broad set of indicators that give a more complete picture of team operation and results. This cuts down on any tendency to gamify the metric, and also leads to a more diagnostically useful view. Metric frameworks like SPACE and Github ESSP bake in this balanced approach, so starting with thoughtful designs like these is good practice.

The implication for organizational observability is to work to achieve an initial baseline of completeness and quality as quickly as practical. Having a balanced cross section of metrics, even if it’s just the basics, and having these be “correct,” at least enough to be useful for understanding how the team is working, means you can show tangible value to a team. This sets you up to achieve the virtuous cycle of continued attention to data quality, and buy-in on future iterations to expand the data and develop an even more comprehensive view.

That said, just because the data is in decent shape and carries valuable insights, it’s far from a guarantee that people across the team will use it to guide actions. Metrics are helpful input for spotting issues and making rational decisions, but they require human interpretation to provide value. A stressed out EM being pushed to quickly adjust some resourcing for a new priority may not be super keen to go consult a bunch of data.

So how do we connect people on the team with the value of metrics?

One proven approach to model the desired behavior is to incorporate metrics into management 1:1s and skip levels. Use team-level metrics as an opportunity for joint curiosity – what seems to be working, and what seems to need improvement? Can we see root causes in the data?

I often asked EMs to preview their team dashboard ahead of time, and I committed to do that as well, which often provided a bunch of interesting topics for discussion. These discussions frequently drove very concrete operational value, and always helped illuminate what was happening on the team to a greater degree. But beyond that, having the discussions was an opportunity to express and demonstrate the value of metrics. And on a very basic level, sometimes these conversations uncovered data issues, which could then be addressed.

As a concrete example, the conversation might start based on looking at a velocity metric being down compared to last month, but there being a clear cut reason why – e.g., the team is in the early stages of a new project and still building clarity on definition; or the team recently grew and is spending extra time onboarding new members. While not uncovering any earth shaking decisions or adjustments, these kinds of cases are opportunities to celebrate that the metrics did actually provide clear visibility to productivity factors, which can and should vary over time based on factors like those described. We didn’t learn anything surprising, but we gain confidence that our telemetry works.

A more interesting example might be a conversation where velocity is down but the reason is not clear. For example, suppose we’ve seen cycle time increase or PR volume decrease in the last quarter, but the EM isn’t sure why, even though they have some qualitative intuitions. For example, they might feel that the team was moving faster last quarter when they were in the heart of a recent exciting project. Clarity was strong, and everyone had a shared sense of urgency. This quarter they’re on to the long tail of polish for that initiative post shipping, and there’s a backlog of small enhancements in older areas that had built up in the push to ship the new feature. Looking at the allocation by initiative we see that WIP is up significantly compared to last quarter. This is quantitative evidence of what the EM is feeling. Can we run an experiment to reduce WIP and possibly get the team more focused? Do we have recent DevEx survey results that might help corroborate our hunch, or perhaps challenge it?

This is an awesome conversation! Might you have had it without metrics? Certainly. But the metrics improve the clarity and context of the discussion. And the value provided to the EM creates a real reason to continue to care about data quality, which then supports additional goals of seeing accurate data about larger trends org-wide over time. It’s an opportunity to model the behavior of managing with metrics for the benefit of the team. The key is that it can and should be a win-win.

Incorporating the use of metrics into day to day management creates a virtuous cycle for data hygiene. But it’s also worth noting that data itself is a powerful tool for inspecting and managing data quality directly.

Issues in the output analytics can clearly point to data and process issues. For example, looking at an allocation report and finding that a significant amount of investment is landing in the “Other” or “Uncategorized” bucket tells us that we have work to do on our issue or epic tagging.

Or in some cases direct reporting on data quality can be helpful. For example, in one environment I managed we encouraged linking PRs and Jira tickets explicitly as part of our data standards, which allowed us to see more accurate allocation and effort reports. But of course, tagging the ticket onto the PR is another thing to remember, and didn’t happen all the time. In our reporting environment, based on Jellyfish, we had access to a chart of unlinked PR rate, which allowed us to identify spots where we needed improvement.

If you’re considering putting some energy into advancing your engineering metrics strategy, perhaps motivated by wanting to understand the impact of AI in your organization, you’re not alone if you’re concerned about your data being “a mess.” But this problem is more tractable than it may seem at first.

Focusing on the minimum viable data requirements as dictated by your metrics strategy can simplify the change management you need to an achievable scope. And with some healthy focus and leadership around the value that metrics create, you can enlist the org to make the needed changes happen.

Driving decision making and management with analytics is hard because at the end of the day, we can all actually make gut decisions without looking at the numbers. And in many cases, our gut calls will be reasonable. But if we can get our org working in a data-informed manner, team members can operate on a higher plane of performance and confidence. Winning sells the hard work, and I bet they won’t want to go back to the more primitive approach.