Perhaps they’ve read some of the wrong-headed posts here about why LLM-generated code doesn’t need to be understandable or maintainable by humans.
Putting aside the undeniable fact that these tools are nowhere near that reliable, in reality, code maintainability matters just as much – if not more – when LLMs are working with it.
First, and hopefully you’ve figured this out by now, “AI”-assisted programming without a good suite of fast-running regression tests is very, very risky. Fast tests have such a huge impact on the cost and the risk of changing code that Michael Feathers defines “legacy code” as code that lacks them.
More teams are discovering that they need to be constantly assessing the “strength” of the automated tests their “AI” assistant generates – they’re notorious for weak tests, and for cheating to get tests passing.
I highly recommend regular mutation testing to check for gaps in your test suites.
Clarity matters, because… well… language models. If I’m asking Claude to add a premium tier to video rentals pricing, but the code’s talking about “vd_prc_1” and “tr_rate_fs”, it hasn’t got much to match on. Concepts need to be clearly signposted and consistent with the language we use to describe our requirements.
Duplication’s a problem, because logic repeated 5x takes up 5x the context, and also models might not actually “spot” the repetition, so there’s a risk of drift.
Complexity’s a big problem. LLMs don’t like complex patterns. Overly complex code is likely to fall outside the data distribution, leading to low-confidence matches and low-accuracy predictions.
And then there’s separation of concerns…
LLMs are trained on a huge amount of code snippets of the Stack Overflow variety that contain little or no modularity. That’s their comfort zone, and code they generate will tend to be like that, too.
The irony is that, while they suck at generating effectively modular code – cohesive, loosely-coupled modules that localise the ripple effect of changes – they also suck at modifying code that isn’t highly modular. The wider the ripple effect, the more code gets brought into play, and the further out-of-distribution the context grows.
In this way, they’ll tend to paint themselves into a corner as the code grows. So we really need to keep on top on modular design.
So, yes, maintainability matters in “AI”-assisted coding. A LOT.
<shameless-plug>
If you think your team could use some levelling up or a refresher on software design principles, my training's half-price if you confirm your booking by Jan 31st. Link in my profile.
</shameless-plug>
In my series on The AI-Ready Software Developer, I propose a set of principles for getting better results using LLM-based coding assistants like Claude Code and Cursor.
Users of these tools report how often and how easily they go off the rails, producing code that doesn’t do what we want and frequently breaking code that was working. As the code grows, these risks grow with them. On large code bases, they can really struggle.
From experiment and from real-world use, I’ve seen a number of things help reduce those risks and keep the “AI” on the rails.
Working in smaller steps
Testing after every step
Reviewing code after every step
Refactoring code as soon as problems appear
Clarifying prompts with examples
Smaller Steps
Human programmers have a limited capacity for cognitive load. There’s only so much we can comfortably wrap our heads around with any real focus, and when we overload ourselves, mistakes become much more likely. When we’re trying to spin many plates, the most likely result is broken plates.
LLMs have a similarly-limited capacity for context. While vendors advertise very impressive maximum context sizes of hundreds of thousands of tokens, research – and experience – shows that they have effective context limits that are orders of magnitude smaller.
The more things we ask models to pay attention to, the less able they are to pay attention to any of them. Accuracy drops of a cliff once the context goes beyond these limits.
After thousands of hours working with “AI” coding assistants, I’ve found I get the best results – the fewest broken plates – when I ask the model to solve one problem at a time.
Continuous Testing
If I make one change to the code, and test it straight away, if tests fail then I wouldn’t need to be a debugging genius to figure out which change broke the code. It’s either a quick fix, or a very cheap undo.
If I make ten changes and then test it, it’s going to take significantly longer, potentially, to debug. And if I have to revert to the last known working version, it’s 10x the work and the time lost.
An LLM is more likely to generate breaking changes than a skilled programmer, so frequent testing is even more essential to keep us close to working code.
And if the model’s first change breaks the code, that broken code is now in its context and it – and I – don’t know it’s broken yet. So the model is predicting further code changes on top of a polluted context.
Many of us have been finding that a lot less rework is required when we test after every small step rather than saving up testing for the end of a batch of work.
There’s an implication here, though. If we testing and re-testing continuously, that suggests that testing very fast.
Continuous Inspection
Left to their own devices, LLMs are very good at generating code they’re pretty bad at modifying later.
Some folks rely on rules and guardrails about code quality which are added to the context with every code-generating interaction with the model. This falls foul of the effective context limits of even the hyperscale LLMs. The model may “obey” – remember, they don’t in reality, they match and predict – some of these rules, but anyone who’s spent more than a few minutes attempting this approach will know that they rarely consistently obey all of them.
And filling up the context with rules runs the risk of “distracting” the LLM from the task at hand.
A more effective approach is to keep the context specific to the task – the problem to be solved – and then, when we’ve got something that works, we can turn our attention to maintainability.
After I’ve seen all my tests pass, I then do a code review, checking everything in the diff between the last working version and the latest. Because these diffs are small – one problem at a time – these code reviews are short and very focused, catching “code smells” as soon as they appear.
The longer I let the problems build up, the more the model ends up wading through it’s own “slop”, making every new change riskier and riskier.
I pay attention to pretty much the same things I would if I was writing all the code myself:
Clarity (LLMs really benefit from this, because… language model, duh!)
Complexity – the model needs the code likely to be affected in its context. More code, bigger context. Also, the more complex it is, the more likely it is to end up outside of the model’s training data distribution. Monkey no see, monkey can’t do.
Duplication – oh boy, do LLMs love duplicating code and concepts! Again, this is a context size issue. If I duplicate the same logic 5x, and need to make a change to the common logic, that’s 5x the code and 5x the tokens. But also, duplication often signposts useful abstractions and a more modular design. Talking of which…
Separation of Concerns – this is a big one. If I ask Claude Code to make a change to a 1,000-line class with 25 direct dependencies, that’s a lot of context, and we’re way outside the distribution. Many people have reported how their coding assistant craps out on code that lacks separation of concerns. I find I really have to keep on top of it. Modules should have one reason to change, and be loosely-coupled to other parts of the system.
On top of these, there are all kinds of low-level issues – security vulnerabilities, hanging imports, dead code etc etc – that I find I need to look for. Static analysis can help me check diffs for a whole range of issues that would otherwise by easy to miss by me, or by an LLM doing the code review. I’m seeing a lot of developers upping their game with linting as they use “AI” more in their work.
Continuous Refactoring
Of course, finding code quality issues is only academic if we don’t actually fix them. And, for the reasons I’ve already laid out – we want to give the model the smoothest surface to travel on – fix them immediately.
And I don’t fix all the problems at once. I fix one problem at a time, again for reasons already stated.
And after I fix each problem, I run the tests again, in case the fix broke anything.
This process of fixing one “code smell” at a time, testing throughout, is called refactoring. You may well have heard of it. You may even think you’re doing it. There’s a very high probability that you’re not.
Clarifying With Examples
Here’s an experiment you can try for yourself. Prepare two prompts for a small code project. In one prompt, try to describe what you want as precisely as possible in plain language, without giving any examples.
The total of items in the basket is the sum of the item subtotals, which are the item price multiplied by the item quantity
In the second version, give the exact same requirements, but using examples.
The total of items in a shopping basket is the sum of item subtotals:
See what kind of results you get with both approaches. How often does the model misinterpret precisely-described requirements vs. requirements accompanied by examples?
It’s worth knowing that code-generating LLMs are typically trained on code samples that are paired with examples like this. When we include examples, we’re giving the model more to match on, limiting the search space to examples that do what we want.
Examples help prevent LLMs grabbing the wrong end of the prompt, and many users have found them to greatly improve accuracy in generated code.
Harking back to the need for very fast tests, these examples make an ideal basis for fast-running automated “unit” tests (where “units” = units of behaviour). It would make good sense to ask our coding assistant to generate them for us, because we’re going to be needing them soon enough.
Putting It All Together
If we were to imagine a workflow that incorporates all of these principles – small steps, continuous testing, continuous inspection, continuous refactoring, clarifying with examples – it would look very familiar to the small percentage of developers who practice Test-Driven Development.
TDD has been around for several decades, and builds on practices that have been around even longer. It’s a tried-and-tested approach that’s been enabling the rapid, reliable and sustainable evolution of working software for those in the know. If you look inside the “elite-performing” teams in the DORA data – the ones delivering the most reliable software with the shortest lead times and the lowest cost of change – you’ll find they’re pretty much all doing TDD, or something very like TDD.
TDD specifies what we want software to do using examples, in the form of tests. (Hence, “test-driven”).
It works in micro-iterations where we write a test that fails because it requires something the software doesn’t do yet. Then we write the simplest code- the quickest thing we can think of – to get the tests passing. When all the tests are passing, then we review the changes we’ve made, and if necessary refactor the code to fix any quality problems. Once we’re satisfied that the code is good enough – both working and easy to change – we move on to the next failing test case. And rinse and repeat until our feature or our change is complete.
TDD practitioners work one feature at a time, one usage scenario at a time, one outcome at a time and one example at a time, and one refactoring at a time. Basically, we solve one problem at a time.
And we’re continuously running our tests at every step to ensure the code is always working. While automated tests are a side-effect of driving design using tests, they’re a damned useful one! And because we’re only writing code that’s needed to pass tests, all of our code will end up being tested. It’s a self-fulfilling prophecy.
Embedded in that micro-cycle, many practitioners also use version control to ensure they’re making progress in safe, easily-reverted steps, progressing from one working version of the code to the next.
Some of us have discovered the benefits of a “commit on green, revert on red” approach to version control. If all the tests pass, we commit the changes. If any tests fail, we do a hard reset back to the previous working commit. This means that broken versions of the code don’t end up in the context for the next interaction. (Remember that LLMs can’t distinguish between working code and broken code – it’s all just context.)
The beauty of TDD is that the benefits can be yours whether you’re using “AI” or not. Which is why I now teach it both ways.
The key to being effective with “AI” coding assistants is being effective without them.
Shameless Plug
Test-Driven Development is not a skill that you can just switch on, whether you’re doing it with “AI” or without. It takes a lot of practice to get the hang of it, and especially to build the discipline – the habits – of TDD.
An alarming number of TDD tutorials aren’t actually teaching TDD. (And the more people learn from them, the more bad tutorials we’ll no doubt see.)
If your team wants training in Test-Driven Development, including how to do it effectively using tools like Claude Code and Cursor, my 2-day TDD training workshop is half-price if you confirm your booking by January 31st.
For over a billion years now, we’ve known that “code-and-fix” software development, where we write a whole bunch of code for a feature, or even for a whole release, and then check it for bugs, maintainability problems, security vulnerabilities and so on, is by far the most expensive and least effective approach to delivering production-ready software.
If I change one line of code and tests start failing, I’ve got a pretty good idea what broke it, and it’s a very small amount of work (or lost work) to fix it.
If I change 1,000 lines of code, and tests start failing… Well, we’re in a very different ballpark now. Figuring out what change(s) broke the software and then fixing them is a lot of work, and rolling back to the last known working version is a lot of work lost.
Also, checking a single change is likely to bring a lot more focus than checking 1,000. Hence my go-to meme for after-the-fact testing and code reviews:
The usual end result of code-and-fix development is buggier, less maintainable software delivered much later and at a much higher cost.
And all things in traditional software development have their “AI”-assisted equivalents, of course.
I see developers offloading large tasks – whole features or even sets of features for a release – and then setting the agentic dogs loose on them while they go off to eat a sandwich or plan a holiday or get a spa treatment or whatever it is software developers do these days.
Then they come back after the agent has finished to “check” the results. I’ve even heard them say “Looks good to me” out loud as they skim hundreds or thousands of changes.
Time for the meme again:
Now, no doubting that “AI”-assisted coding tools have improved much in the last 6-12 months. But they’re still essentially LLMs wrapped in WHILE loops, with all the reliability we’ve come to expect.
Odds of it getting one change right? 80%, maybe, with a good wind behind it. Chances of it getting two right? 65%, perhaps.
Odds of it getting 100 changes right? Effectively zero.
Sure, tests help. You gave it tests, right?
Guardrails can help, when the model actually pays attention to them.
External checking – linters and that sort of thing – can definitely help.
But, as anyone who’s spent enough time using these tools can tell you, no matter how we prompt or how we test or how we try to constrain the output, every additional problem we ask it to solve adds risk.
LLMs are unreliable narrators, and there’s really nothing we can do to get around that except to be skeptical of their output.
And then there are the “doom loops”, when the context goes outside the model’s data distribution, and even with infinite iterations, it just can’t do what we want it to do. It just can’t conjure up the code equivalent of “a wine glass full to the brim”.
And the bigger the context – the more we ask for – the greater the risk of out-of-distribution behaviour, with each additional pertinent token collapsing the probability of matching the pattern even further. (Don’t believe me? Play one at chess and watch it go off that OOD cliff.)
So problems are very likely with this approach – which I’m calling “prompt-and-fix”, because I can – and finding them and fixing them, or backing out, is a bigger cost.
What I’ve seen most developers do is skim the changes and then wave the problems through into a release with a “LGTM”.
One more time:
This creates a comforting temporary illusion of time saved, just like code-and-fix. But we’re storing up a lot more time that’s going to be lost later with production fires, bug fixes and high cost-of-change.
One of the most important lessons in software development is that what’s downstream of present you is upstream of future you – as Sandra Bullock and George Clooney discovered in Gravity.
The antidote to code-and-fix was defect prevention. We take smaller steps, testing and reviewing changes continuously, so most problems are caught long before finding, fixing or reverting them becomes expensive.
I have a meme for that, too:
The equivalent in “AI”-assisted software development would be to work in small steps – one change at a time – and to test and review the code continuously after every step.
Sorry, folks. No time for that spa treatment! You’ll be keeping the “AI” on a very short leash – both hands on the wheel at all times, sort of thing.
The other benefit of small steps is that they’re much less likely to push the LLM out of its data distribution. Keeping the model in-distribution more, so screw-ups will happen less often – while reaping the benefits of immediate problem detection in reduced work added or lost when things go south – is a WIN-WIN.
I know that some of you will be reading this and thinking “But Claude can break a big problem down into smaller problems and tackle them one at a time, running the tests and linting the code and all that”.
Yes, in that mode, it certainly can. But every step it takes carries a real risk of taking it in the wrong direction. And direction, despite what some fans of the technology claim, isn’t an LLM’s strong suit. Remember, they don’t understand, they don’t reason, they don’t plan. They recursively match patterns in the input to patterns in the model and predict what token comes next.
Any sense that they’re thinking or reasoning or planning is a product of the Actual Intelligence they’re trained on. It may look plausible, but on closer inspection – and “closer inspection” is often the problem here – it’s usually riddled with “brown M&Ms”.
So, no, you can’t just walk away and let them get on with it. If they take a wrong turn, that error will likely compound through the rest of the processing.
Think of what happens in traditional software development when a misunderstanding or an incorrect assumption goes unchecked while we merrily build on top of that code.
I’ve watched a lot of people using “AI” coding assistants, and noted how often they wave through large batches of code changes the model proposes, sometimes actually saying it out loud: “Looks good to me”.
After nearly 3 years of experimentation using LLMs to generate and modify code, I know beyond any shadow of a doubt that you need to thoroughly check and understand every line of code they produce. Or there may be trouble ahead. (But while there’s music… etc)
But should I be surprised that so many developers are happily waving through code in such a lackadaisical way? Is this anything new, really?
I’ve watched developers check in code they hadn’t even run. Heck, code that doesn’t even compile.
I’ve watched developers copy and paste armfuls of code from sites like Stack Overflow, and not even pause to read it, let alone try to understand it or even – gasp – try to improve it.
I’ve watched developers comment out or delete tests because they were failing. I’ve watched teams take testing out of their build pipeline to get broken software into production.
We’ve been living in an age of “LGTM” for a very long time.
What’s different now is the sheer amount of code being waved through into releases, and just how easy “AI” coding assistants make it for the driver to fall asleep at the wheel.
And when we put our coding assistant into “agent” mode – or, as I call it, “firehose mode” – that’s when things can very quickly run away from us. Dozens or hundreds of changes, perhaps even happening simultaneously as parallel agents make themselves busy on multiple tasks at once.
Even if there were no issues in any of those changes – and the odds against that are extremely remote – when code’s being churned out faster than we’re understanding, it creates a rapidly-growing mountain of comprehension debt.
When the time comes – or should I say, when the times come – that the coding assistant gets stuck in a “doom loop” and we have to fix problems ourselves, that debt has to be repaid with interest.
Agents have no “intelligence”. They’re old-fashioned computer programs that call LLMs when they need sophisticated pattern recognition and token prediction. LLMs don’t follow instructions or rules. Use them for just a few minutes and you’ll see them crashing through their own guardrails, doing things we’ve explicitly told them not to do, and forgetting to do things we insist that they should.
The intelligence in this set-up is us. We’re the ones who can follow rules and instructions. We’re the ones who understand. We’re the ones who reason and plan. And we’re the ones who learn.
In 2025, and probably for many years to come, we are the agents. We’re the only ones qualified for the job.
My advice – based on the best available evidence and a lot of experience using these tools over the past 3 years – remains the same when you’re working on code that matters.
I recommend working one failing test at a time, one refactoring at a time, one bug at a time, and so on.
I recommend thoroughly testing after every step, and carefully reviewing the small amount of code that’s changed.
I recommend committing changes when the tests go green, and being ready to revert when they go red.
I recommend a fresh context, specific to the next step. I recommend relying on deterministic sources of truth – the code as it is (not the model’s summary of it), the actual test results, linter reports, mutation testing scores etc.
I strongly advise against letting LLMs mark their own homework or rely on their version of reality.
And forget “firehose mode” for code that matters. Keep it on a very tight leash.
The idea of “separation of concerns” originated from a need to make it possible for programmers to reason about a piece of code without the need to understand what’s going on inside its dependencies (and its dependencies’ dependencies).
In this sense, the primary benefit of modular design is to reduce cognitive load when working with any part of the system.
But that can only happen if every reference to other parts (e.g., function calls) “says what it does on the tin”, so we can form correct expectations about its behaviour within the context we’re reasoning about.
Ideally, to understand what a dependency does, we shouldn’t need to understand how it does it.
When names are unclear, or even misleading, we form the wrong expectations about what a dependency will do, and are forced to “look inside the box” to understand it.
In the same way that this increases the context size for an LLM and therefore the risk of errors, it increases cognitive load for programmers with the same end result. I see folks complaining often about having to have a bunch of source files open in order to understand what one piece of code is going to do.
It might be helpful here if I put forward a definition of code comprehensibility and a rough way of measuring it.
I see comprehensibility as the likelihood that the target audience (e.g., other team members) will correctly predict what a piece of code will do in specific cases.
What will the value of average_rating be after that assignment?
Let’s refuctor that code to make it a little less obvious.
r = [4, 6, 4, 5]
ar = s(r)/l(r)
Now you need to know what the functions s and l do. That may mean looking it up, if there’s documentation available – more cognitive load.
Or it may mean actually peeking inside their implementations – even more cognitive load.
We could ask 10 developers to predict what the result will be. If 8 of them predict correctly, we might roughly gauge the comprehensibility of this code for that sample of people – remember who the audience is – is 80%.
Or we could ask an LLM ten times. Though it’s important to remember that LLMs can’t reason about behaviour. They would literally just be matching patterns. And in that sense, this is a reasonable test of how closely the code correlates with examples within the training data distribution.
So, in summary, naming is very, very important in modular software design. Names help us form expectations about behaviour, and if those expectations are correct then this means we don’t need to go beyond a signpost to understand what’s down that road.
This mantra is the answer to the question “When should we do code reviews?” We do them whenever we see the software working again – whenever the tests are green. (You have tests, right?)
In the TDD cycle, we start by defining what we want the software to do by writing a failing test. Then we do the simplest, quickest thing to get the tests passing.
I stress in my training courses that this is not the time to be agonising over the design. Priority #1 – MAKE IT WORK!
When all the tests are passing, then we have that luxury. We can take a step back and review the code we’ve added or changed – including the test code – and ask ourselves if there’s anything about it that’s going to make changing it later harder than it needs to be.
We consider readability, complexity, duplication, coupling and cohesion. We might even run a linter over it to check for low-level issues we might have missed.
We get the tests passing, and we do our code review. We do it when we go from red->green in the TDD cycle. We do it after every refactoring to validate the end result.
“That sounds like a lot to be doing on every green light, Jason!”
That depends how far apart your green lights are. The tighter the feedback loop, the smaller the diff, the quicker the code review. And you might be surprised at how much of it can be automated, if you’re willing to invest some time.
More than one pair of eyes can really help, too.
The payoff is three-fold:
1. No code review bottleneck when you want to ship
2. Catching design problems straight away means they’re often much cheaper to fix (or to back out of)
3. Reviewing code in micro-batches enables much greater attention to detail – fewer problems slip through the net. How does the saying go? “Show a developer a line of code and they’ll tell you what’s wrong with it. Show them 500, and they’ll say ‘Looks good to me’ “
The side-benefit is that performing systematic code reviews over and over will tend to bake them into your subconscious, turning you into a human linter. You’ll develop, as Keith Braithwaite once put it to me, “good taste in code”.
90% of the time, TDD is enough to assure that code of the everyday variety is reliable enough.
But some code really, really needs to work. I call it “load-bearing code”, and it’s rare to find a software product or system that doesn’t have any code that’s critical to its users in some way.
In my 3-day Code Craft training workshop, we go beyond Test-Driven Development to look at a couple of more advanced testing techniques that can help us make sure that code that really, really needs to work in all likelihood does.
It raises the question, how do we know which parts of our code are load-bearing, and therefore might warrant going that extra mile?
An obvious indicator is critical paths. If a feature or a usage scenario is a big deal for users and/or for the business, tracing which code lies on the execution path for it can lead us to code that may require higher assurance.
Some teams work with stakeholders to assess risk for usage scenarios, perhaps captured alongside examples that they use to drive the design (e.g., in .feature files), and then when these tests are run, use instrumentation (e.g., test coverage) to build a “heat map” of their code that graphically illustrates which code is cool – no big deal if this fails – and which code might be white hot – the consequences will be severe if it fails.
(It’s not as hard to build a tool like this as you might think, BTW.)
A less obvious indicator is dependencies. Code that’s widely reused, directly or indirectly, also presents a potentially higher risk. Static analysis tools like NDepend can calculate the “rank” of a method or a class or a package in the system (as in, the Page Rank) to show where code is widely reused.
Monitoring how often code’s executed in production can produce a similar, but dynamic, picture of which code’s used most often.
These are all measures of the potential impact of failure. But what about the likelihood of failure? A function may be on a critical path, and reused widely, but if it’s just adding a list of numbers together, it’s not very likely to fail.
Complex logic, on the other hand, presents many more ways of being wrong – the more complex, the greater that risk.
Code that’s load-bearing and complex should attract our attention.
And code that’s load-bearing, complex and changing often is white hot. That should be balanced by the strength of our testing. The hotter the code, the more exhaustively and the more frequently it might need testing.
Hopefully, with a testing specialist in the team, you will have a good repertoire of software verification techniques to match against the temperature of the code – guided inspection, property-based testing, DBC, decision tables, response matrices, state transition tables, model checking, maybe even proofs of correctness when it really needs to work.
But a good start is knowing where your hottest code actually is.
At the moment, I offer 5 “off-the-shelf” training workshops focused on the core technical practices that enable rapid, reliable and sustained evolution of working software to meet changing needs.
Basically, the practices that have been shown to reduce delivery lead times, while improving release stability and reducing cost of change.
They’re self-supporting (e.g., can’t have continuous testing without good separation of concerns) – so ideally, your team would apply all of them in a “virtuous circle”.
But when I look at the sales history of each workshop, there’s a worrying imbalance.
* Code Craft (the flagship workshop) sells 49% of the time.
* The 2-day introduction to Test-Driven Development, aimed at less experienced developers, sells 32% of the time.
* The 1-day introduction to Unit Testing sells 9% of the time.
* And the 2-day Refactoring deep-dive only 2%. In fact, nobody’s booked a refactoring workshop since before the pandemic!
Refactoring, as a skill, exercises many of the “muscle groups” involved in Continuous Delivery, and is one of the most challenging to learn.
It’s also one of the most valuable. Whether you’re doing TDD or not, whether you’re continuously integrating or not, whether you’re agile or not – the ability to safely and predictably reshape code to accommodate change is gold.
Without it, you are far more likely to break Gorman’s First Law of Software Development:
Thou shalt not break shit that was working
Especially when you consider that most developers are working on hard-to-change legacy code most of the time. Refactoring is the skill for working with legacy products and systems.
I promised I wouldn’t be mentioning it this week, but I’ll just subtly hint that this problem is currently accelerating because of… well, y’know.
I routinely cite it as the second most important software development skill. (Can you guess what I believe is the first?)
It’s ironic, then, that it’s one of the rarest and one of the least in-demand, if job specs and training orders are any indication.
For sure, most developers will use the word (typically not knowing what it means), and most developers will claim they do it. But the large majority have never even seen it being done – hence the many misapprehensions about what it is.
At the very least, it would be a step-change for the profession if the average software developer could recognise the most common “code smells” and had a decent set of primitive refactorings in their repertoire to deal with them. I call this “short-form” refactoring.
And ideally, a good percentage of us would be capable of “long-form” refactoring so we can reshape architecture at a higher level safely. The best software architects have learned to think that way. (e.g., Joshua Kerievsky’s excellent book Refactoring To Patterns).
If you’d like to build your team’s Refactoring Fu, visit the website for details.
The 2025 DORA State of AI-Assisted Software Development report shows a trend where teams that were already high-performing appeared to make modest improvements in delivery lead times and release stability using “AI” coding assistants, while teams that were less than high-performing showed noticeable losses as the code-generating firehose overwhelmed the bottlenecks in their system.
The question on my mind today, after spending time this morning with a team I’ve worked with for several years, is whether this is cause or correlation.
It struck me, several months after our last coaching sessions, that they were running their tests more often, committing more often, and being more systematic about code inspections in today’s session.
And they’ve been mutation testing diffs to make sure all the code is needed, and if it is, it’s meaningfully tested.
They’re also linting locally for low-level issues like floating imports and unused declarations, whereas before they let the build handle that through SonarQube. (“Horse”, “Stable Door”, etc).
Basically, they appear to have tightened up their feedback loops.
When I complimented them on this, they said that they’d had to do it because the firehose – which management had mandated after our previous session – was pushing their delivery and quality metrics in the wrong direction.
Which raises the thought in my mind: is it the “AI” making them marginally more productive directly, or is it indirectly because the “AI” is forcing them to be more disciplined?
This would then beg the question, why doesn’t it have the same positive effect on the discipline of the other teams – the low-performing and “Meh” teams?
One possible explanation is that they’re not paying attention to the bottlenecks, the delays and the cruft to anywhere near the same extent – “LGTM”.
They haven’t noticed that the car’s slowing down because they’re not looking at the dashboard.
And if they’d had an incentive to improve, they would have improved already. So why would they start now?
Software development becomes a distinctly different game at different scales.
What might be fine for a proof of concept with maybe just a few hundred lines of code is likely to bring the whole house tumbling down at tens or hundreds of thousands of LOC.
When evaluating advice about approaches to development, be careful to find out at what scale it’s been tested.
For example, products like JUnit and iPlayer demonstrate Test-Driven Development at an appreciable scale, and over many years.
As a result, I have high confidence in TDD on a wide range of problem types. Some might think it’s overkill for a proof of concept, but over more than 25 years, it’s proven itself valuable at larger scales in terms of its impact on delivery lead times, product reliability and sustaining the pace of development.
I think this is probably especially important with claims made about “AI”-assisted coding, because there’s a lot advice out there from people who only seem to have used it on relatively small, single-person projects, and very few substantial code bases maintained by teams that we’ve seen stand the test of time (yet).
In this sense, “AI” coding assistants – and the techniques we’re discovering tend to produce better results with them – are like a new drug that’s still very much in testing, and the long-term side-effects may not show for years to come. (Though we are definitely seeing some short-term side-effects!)
I’m trying as much as possible to take a measured, evidence-based approach to using the technology. Wherever possible, I try to see it applied at appreciable scales that are more representative of the code we’re typically working with, and build on studies that are more towards the credible end of the spectrum. But there are few large-scale – and no long-term – studies to guide us here.
“AI” could turn out to be our aspirin, or it could turn out to be our cocaine – something we’ll have to spend countless billions dealing with the downsides of in the future.
But when we’re evaluating advice about using “AI” coding assistants, we should consider the scale at which it’s been tested. And we should consider if the person offering the advice even looked for side-effects, or recognised them when they saw them. Maybe their bar is set differently to ours.
Codemanship's Blog 