Technical Debt

Technical debt is, in a simple way of framing it, a part of your systems that slows you down. Often the debt exists as a result of an explicit choice -- either trade-offs to accelerate in the short-term, or the prioritization of new development over maintenance work. The lines can be blurry. Debt makes for a particularly apt metaphor, for as in it does in finance, the costs grow over time, and can compound with confounding factors. Managing debt is a critical component of a healthy organization, and you as an individual should feel empowered to make improvements along the way.

Debt has the ability to slow you down in ways that are difficult to predict, measure, or counter.

Fragile Environments are those that are difficult to change without breakage, and can take a number of forms. Perhaps you have an application that has few (or no) tests. Changes in these types of environment are simple to the casual observer, and even to a naive first attempt. Those attempts can leave behind non-obvious breakages, which may hurt your credibility or cripple your products. The environments will, over time, shatter your confidence in making changes safely, as well they should. The confidence problems will translate into longer time spent making changes, and the faults (inevitably) introduced will drain time from your team.

Added Complexity, even in a stable environment, makes change difficult. Modifying a system that is complex (and often coupled), requires loading most (if not all) of the system into one's head, a skill that takes time and is not simply transferrable amongst people or systems. Michael Lopp (of Rands in Repose) has a shirt design that captures this succinctly.

External Commitments, the most basic of which is "I will keep this thing working", can be difficult to work around. Sometimes, your users will assume your current behavior is the specification, even if your actual specification explicitly calls that out as untrue. A great example of this is Stripe's API consistency semantics. As of this writing, Stripe's APIs generally support list-your-writes consistency, which is a tier above read-your-writes consistency, in that it requires that not only the data itself, but all indexes of that data be strongly consistent. This is a huge pain! Even if all of the internal use-cases migrated to eventually consistent indexes, external users, and the applications and platforms they leverage, depend on this behavior. If you are unable or unwilling to break those users, then this is forever.

Starting from zero (or, less likely, an existing zero-debt system), the process starts with the introduction of debt. This can occur for a variety of reasons, most of them related to a desire to reduce costs (through shorter development time, less-costly development resources, cheap off-the-shelf solutions, etc). This phase is the least-felt by all parties -- those parties prioritizing velocity probably got what they wanted. Those involved in execution likely felt minimal pain (existential angst in tarnishing a good thing, aside), and expended effort roughly in line with estimates.

In early stages, debt can be mitigated through avoidance -- there's still room enough to in the product real-estate to build shiny new things (with a mixture of debt policies). Occasionally, a new feature will stumble up against an area of known (or forgotten) debt, and can nimbly pivot around it. This pivot is itself a form of debt, creating more debt-y surface area. Over time, these combine, making the occasional stumble the norm.

Members of the organization will share stories of "that bad place in the system", which will influence development and innovation. If a designer knows that tool-tips are particularly brittle in the product, they'll craft potentially less-good designs to circumvent. If a manager knows that introducing changes to the backend will introduce significant delays, they may choose to give up features like "shared persistence" to pursue client-only solutions.

A system that is about to grind to halt has come up against obstruction -- it is no longer possible to avoid debt, and the debt encountered actively prevents you from making forward progress. Seemingly simple changes (like changing the ordering of a list) could require weeks of effort. Those weeks pale in comparison to the effort estimated for addressing the mountain of debt ahead of you, and it becomes increasingly difficult to justify tackling the debt. Obstruction can manifest in glacial delivery of new features, in stalled or failed infrastructure migrations, and everything in between.

It's not uncommon for the phrase "ground-up rewrite" to start popping up at this phase, but it's a catastrophic mistake. A system in this state is unlikely to be well understood, and building a replacement without that understanding is not set up for success. I came across an article by Herb Caudill a couple years back, and shared it with my engineering leadership with respect to the N-th rewrite attempt the organization was undertaking (with N-1 failures under out belt). I was particularly taken with the story of FreshBooks & BillSpring as a clever circumvention of debt -- fully spinning out a competitive product, and allowing it to live or die based on its merits in the marketplace. It's arguably a bit cut-throat, but if you have kind and reasonable leadership in place, it would be an interesting exercise to build anew, sunset the old, and be the one turning out the lights on the past product.

While the first known use of the debt metaphor came from Ward Cunningham via OOPSLA '92, I'd recommend reading Fowler's' and Apenwarr's thoughts in this space. There's also a myriad of writings on the internet about "fixing technical debt", many of which are about using process to prioritize and keep debt on the minds of decision makers. I'm not particularly a fan of these, as it assumes a (disappointing) degree of agency in individual contributors.

During my tenure in AWS, someone explained to me their (interpretation of) the philosophy around having engineers be on-call for services.

Nothing gets something fixed like annoying an engineer twice.

This can be polished a bit, clarifying why it's true.

Problems are solved by those who are motivated to and capable of fixing the problem.

What's missing from the above is any mention of opportunity -- a manager or a planning partner authorizing you to address that technical debt. Years ago, Randall Munroe (of xkcd) published a tongue-in-cheek-but-actually-useful chart to help visualize the value of your time spent improving a task.

This chart is self-centered, in that it assumes you're the only person that will benefit from changes. It provides a useful lower bound on the potential impact of any improvements you might make, but the reality is much more interesting. If you're part of a team (and a builder of tools), you can likely create something that improves the task for most people. The question then becomes, how much of your time are you willing to spend? Losses are usually localized to your own time, which you can budget pretty easily. Use your judgment.

There are likely entire branches of mathematics, subreddits, and self-help gurus dedicated to deciding whether to invest or simply pay interest on the debt. You should feel free to consume of much of that material as you can hold in your head, however I have a much simpler process.

If it's less than one day, just do it.

That was the simple mantra 23-year-old me came up with, and I was lucky to stumble into a couple of early wins. The most valuable thing that those wins provided me with was credibility; an opportunity to make mistaken investments and walk away largely unscathed. But the second most valuable thing those early successes gave me was time. Time that could be reinvested. Time that created more time.

Nowadays, my philosophy is a bit more nuanced; I'll account for the urgency of several in-flight projects, my perceived probability of success (or at least resulting in a net timing payoff for the urgency minded), and the potential for broader impact beyond myself. Another aspect I'll pull in is more psychological -- time isn't necessarily the whole consideration. Maybe that 5-minute task I have to do once per month will take a week to automate, but I loathe the task so much that it's worth it to me personally to never do it again. Maybe I just really like the project and need to scratch an itch -- I have a homebrew voice control system in my office that runs ASR and TTS from a single RaspberryPi, and the amount of time I spent compiling the correct version of PyAudio from source so that my office greets me with insults from a video game will never work out in any kind of spreadsheet. When in doubt, I'll fall back to the simple version.

If it's less than one day, just do it.

Spend some time, earn some time back. Repeat the process. Use your judgment. If your judgment is bad, listen to feedback.

Systems that have reached a state of oppressive technical debt are hurting you in two ways: unexpected failures are eating your time and complexity is resistant to change. Over an extended period of time, you've also likely lost context on your team, making complex systems even more opaque. To make forward progress, you need to incrementally address all three.

Towards the end of my tenure at Stripe, a project was sunset and I was placed with a team that newly owned a legacy and very critical part of the product: payments. The team was coming off a pretty big let-down: a ground-up rewrite had been scrapped, and they were transitioning from a variety of new platforms and technology to the legacy stack. There were some old hats in the mix (much to my pleasure), but many folks had never written software in the legacy language (ruby), let alone on the internal platforms supporting it. A plan was conceived.

In phase 1, with the help of folks who knew the old ways, we seeded the team with a series of optimization projects -- some low-hanging fruit, some in the bowels of the codebase. The optimization projects were somewhat resilient to shifting winds; they could be categorized as product improvements or cost savings measures, and in several cases were unblocking future product development. This helped prevent work from being deprioritized and shuffled, and was critical to success of the plan. The folks newer to the space picked projects and ran with them, with the more tenured folks providing support and context when possible. In this phase, the team gained context and confidence building in this critical area.

In phase 2 (and realistically, along the way in phase 1), we put out the suggestion box. New project ideas, big and small, came in from the rest of the team. We taught them our prioritization philosophy; optimization touches on many dimensions, and we'll try to strike a balance that's resilient to shifts in broader priorities. As a general principle, take small and medium projects from start to finish.

In phase 3, the team has context, has confidence, and has built up a backlog of work they want to do. From that backlog, we tease out themes, and craft some grander proposals for the future of the space. The entire team is confident handling on-call, and slowly but surely, things are improving.

The down-side to this plan was that it didn't lend itself (from the outset) to a long-term vision. Our leadership (at most levels) wanted to know what the plan was, multiple years in the future. While I could point to the success of the team, and could make credible arguments against the value of long-term plans made without context, the absence of a lengthy roadmap was unpopular. I don't believe that roadmap exists yet (or that it would have survived first contact with reality), but the team is much healthier.

Slow, incremental progress is not an engaging organizational strategy. It is, however, an effective one.