Codemanship's Blog

AI Software Development – What Does The Data Say?

I’m currently pulling together a bunch of sources – that are mostly recent – on the topic of LLMs and their use in software development.

Some are peer-reviewed studies. Some are industry studies that haven’t been peer-reviewed.

One is statistical physics. Expect more from that angle. Wanna’ know the limits of a technology? Ask a physicist.

One is just a blog post, but very useful information about the effect of context size.

Most are corroborated by personal experiments and also observations on teams. As time goes on and more data comes in, my picture comes more into focus.

Before I cite the sources, a quick executive summary for all you busy executives out there:

Truly autonomous and reliable long-horizon agentic software development is so highly improbable using LLMs that it’s essentially science fiction.
The maximum effective context limits of LLMs – including hyperscale “frontier” LLMs – beyond which model outputs become unusably inaccurate is orders of magnitude smaller than advertised limits. The most common mechanism for extending inference over large contexts is what vendors call “compression”. This means that parts of the context are summarised by the model, which is a famously unreliable/lossy process.
LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training (“dominant priors”), can often “outweigh” information we give it. To an LLM, it’s all just tokens, weights and probabilities. Right, wrong, new, old – the highest probability wins. Big contexts and “attention dilution” – where probabilities in the context become too small to compete with the ones in the model – are likely to make these effects worse.
Repo-level .md files tend to make model performance worse, probably because they add noise instead of signal in many specific tasks. Model-generated .md files are especially problematic in this respect, it seems. Upshot: including your team’s coding standards and an architecture summary for every task is probably counterproductive.
LLMs struggle with negation. Telling them not to do something can often have the same effect as telling them to do it. In case you were wondering why some of your guardrails are about as reliable as a coin-toss.
LLM inference is more accurate when we give them examples (demonstrations) rather than just describing what we want. They’re pattern-matchers. Show them the patterns – more “like this” and less “do this” (and no “don’t do this”).
Large/long-scale industry studies show a clear trend – output is up (more code, more commits, bigger diffs), but outcomes don’t reflect that trend. If anything, the average team is taking longer to ship worse software. If ever we needed proof that software development isn’t a production process… Some studies find a small % of teams getting modest gains in outcomes, and correlate that with their existing software development capability. AI coding is an amplifier of, not a fix for, development strengths and weaknesses. (You’d think organisations would be lining up to do something about that… Sigh.)
The psychological and cognitive factors in LLM use are a growing field of serious research. One study found a significant correlation between confidence in AI output and belief in the paranormal. Multiple studies found a negative impact on learning, cognition and critical thinking with greater LLM reliance. New research suggests that reports of developers feeling demotivated and burned-out with extensive use may have some real truth behind them.
Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model. They will always be “driving in fog”, with local, short-range probabilities crowding out long-range ones. In case you were wondering why they suck at the “big picture” – probabilistically, it’s a blur.
The energy and compute needed to train an LLM to be an order of magnitude more reliable – e.g., wrong 3% of the time instead of 30% – is 10^20 times what the current frontier models require. Don’t expect significantly more reliable models any time soon. Any future gains in reliability will have to made by better context engineering (deciding what to include in the input) and more effective quality gates deciding what to do with the output- and that’s exactly what we’re seeing AI companies focusing on these days. Models may get more powerful, but not significantly more reliable. This it folks – work with what you’ve got!
Some AI champions will protest research that points to no significant improvements in model performance by pointing to the many published benchmarks that do indeed show LLMs getting better and better. But other research finds that we might wish to be more skeptical of benchmark performance, partly because many of the most popular ones measure what’s easy to measure algorithmically – and in that sense, they’re not really like real-world problems which are messy and unpredictable – and also because… well, the words “published benchmarks” are a bit of a clue. Perhaps inadvertently, but maybe even knowingly, increasingly models are being “trained to the test”. If it’s out there, then it’s probably in there.

Code Evolution & Long-Horizon Agentic Workflows

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
https://arxiv.org/abs/2603.03823

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
https://arxiv.org/html/2603.24755v1

SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
https://arxiv.org/abs/2603.13428

Benchmark vs. Real-World Performance

Measuring what Matters: Construct Validity in Large Language Model Benchmarks
https://arxiv.org/abs/2511.04703

Research Update: Algorithmic vs. Holistic Evaluation (METR – not peer-reviewed)
https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/#background

Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
https://arxiv.org/abs/2411.03923

Context Engineering

Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs
https://arxiv.org/abs/2509.21361

Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
https://arxiv.org/abs/2509.21865

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
https://arxiv.org/abs/2602.11988

(blog post – informational, not peer-reviewed research)
The Hidden Science Behind LLM Token Limits (And How Million-Token Models Actually Work)
https://www.ashisharora.ai/post/the-hidden-science-behind-llm-token-limits-and-how-million-token-models-actually-work

Language models are not naysayers: An analysis of language models on negation benchmarks
https://arxiv.org/abs/2306.08189

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
https://aclanthology.org/2022.emnlp-main.759

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
https://arxiv.org/abs/2605.06527

Task Matters: Knowledge Requirements Shape LLM Responses to
Context–Memory Conflict
https://aclanthology.org/2026.findings-acl.202

(I call these “dominant priors”, when information in the model overrides that provided in the context)

Large-scale Industry Studies in Software Engineering

What 28 million workflows reveal about AI coding’s biggest risk (CircleCI)
https://www.linkedin.com/pulse/what-28-million-workflows-reveal-ai-codings-biggest-risk-circleci-j9syc/

The Acceleration Whiplash – AI Engineering Report 2026 (Faros)
https://www.faros.ai/research/ai-acceleration-whiplash

State of AI-assisted Software Development 2025 (DORA)
https://dora.dev/research/2025/dora-report/

Psychology, Cognition & Learning

Super-intelligence or Superstition? Exploring Psychological Factors Influencing Belief in AI Predictions about Personal Behavior
https://arxiv.org/html/2408.06602v3

The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers
https://www.researchgate.net/publication/391270185_The_Impact_of_Generative_AI_on_Critical_Thinking_Self-Reported_Reductions_in_Cognitive_Effort_and_Confidence_Effects_From_a_Survey_of_Knowledge_Workers

Experimental evidence of the effects of large language models versus web search on depth of learning
https://www.researchgate.net/publication/397000021_Experimental_evidence_of_the_effects_of_large_language_models_versus_web_search_on_depth_of_learning

At What Cost? Software Developers’ Well-Being in the Age of GenAI
https://ourarchive.otago.ac.nz/esploro/outputs/preprint/At-What-Cost-Software-Developers-Well-Being/9926870122801891

Limits of LLMs & Deep Learning

The wall confronting large language models – (statistical mechanics study)
https://arxiv.org/abs/2507.19703

Learning long-term dependencies with gradient descent is difficult
https://pubmed.ncbi.nlm.nih.gov/18267787/
(Why deep neural networks can’t see the bigger picture, at any scale of model)

What are the implications for teams using this technology? If you’re interested in a hype-free, evidence-based take about why the technical practices of Agile Software Development are so closely aligned with AI-assisted and agentic software engineering, join me on October 6th at 18:45 BST.

Register: https://www.tickettailor.com/events/codemanship/2324138

Do Your Quality Gates See The Brown M&Ms In The Bowl?

I see more and more converts to mutation testing, as folks learn that extending the horizon of agentic coding requires much, much stronger quality gates.

Mutation testing deliberately injects errors into code to create “mutant” versions – like turning a + into a -, or replacing a string with “” – so we can see if any of our automated tests catch the error. If no tests fail and the mutant survives, that reveals a potential gap in the test suite.

Mutation testing is a specialised version of what I’ve been calling “Brown M&M tests”. Perhaps you know the story – Van Halen had a clause buried in their venue rider that there should be a big bowl of M&Ms in the dressing room with all the brown ones taken out.

Not an example of diva-like rock start excess, but a very practical test of whether the venue had paid attention to every detail. Van Halen’s live show had many complex technical elements. If they entered their dressing room and saw brown M&Ms in the bowl, they double-checked everything.

The rise of mutation testing’s encouraging and frankly long overdue. But don’t stop there!

How are you testing your linter rules? I might deliberately inject unused imports into random source files to see if my automated code review finds them all. I might deliberately inject security vulnerabilities, or race conditions, or nonsense identifier names – all example of “brown M&Ms” – so I can test for gaps in those quality gates.

(Now, go on, admit it, “dark factory” folks, this probably never occurred to you, did it?)

I realised – through experiment and research going back 3+ years – that people’s confidence in LLM-generated code has more to do with whether or not they see the “brown M&Ms in the bowl” than the actual code’s quality.

When I see these claims of high-quality generated code that I’ve not been able to reproduce, I’ve stopped asking “What are they doing that I’m not doing?” – at this point I’m way ahead of the curve – and now ask “What am I seeing that they’re not seeing?”

Join me for a hands-on, evidence-based workshop – free of hype and hopium – on Code Craft & AI. Tues Oct 6th 18:45 BST. Just £99 + VAT for self-funding learners.

The Power Of “Why?”

How many times in your career as a software developer have you been handed a “requirement” that turned out to be difficult and costly to fulfil?

I’ve been there many times, and I imagine you have, too.

The hack I discovered a few years into my career is that one person’s “what” is almost always another person’s “how”. Software requirements very rarely describe an actual need – a user goal, for example. They’re usually a decision that someone has taken about how to address a need.

When we make decisions – the user will select their house number from a drop-down list – we constrain future possible decisions, reducing the options we have available to solve a problem we might not even be aware exists.

Visualising design decisions as a tree, we can more clearly see the relationships between decisions – how one decision is a consequence of an earlier decision.

We can traverse the tree in three directions – we can go down by asking “How?” How are we going to populate the list of house numbers? I call this direction of traversal “execution”.

We can go left or right by exploring other options – does it have to be a drop-down list? Could it be they stick a pin in a street map instead? Or just enter it in a text box? I call this direction of traversal – where we stay at the same level in the tree and evaluate other paths – “exploration”.

And we can climb back up the tree to the root decision that led us to these options by asking “Why?” Why does it need to be selected from a drop-down list? So we can be more confident it’s a real address. Okay, so why do we need to be confident it’s a real address? Because we’ll be delivering to that address.

And there, folks, is Waldo! That’s the real need.

Traversing the decision tree in this direction helps us understand not just what we’re creating, but why we’re creating it. It’s the context that’s so often missing in the design process.

Why is “Why?” such a powerful question?

Remember that in a decision tree, each decision constrains possible future decisions. Each time we commit to a specific branch, we lose significant optionality – we get fewer execution choices to work with and a smaller potential solution space. This reduces our chances of solving a problem with limited time and resources.

When we take a step back to an earlier root decision, optionality opens up again. I’ve lost count of the times when a deadline was looming and the path we’d chosen just wasn’t going to get us there in time, and I asked the question “What is it we’re trying to achieve, again?” 99% of the time, it turned out the requirement wasn’t the real requirement – it was a decision. And 99% of those times, it turned out there were quicker and cheaper options – often not involving writing any code at all.

In fact, this happened so often in the early part of my contracting career that I eventually adopted a policy of questioning every requirement in a specification, and ultimately rejecting any specification the team hadn’t been directly involved in agreeing.

This is why I refer to software requirements specifications these days as “handcuffs”.

And we must not forget that, no matter how clear the rationale for a design decision is, it is at best an educated guess at what will solve the root problem. Only reality can answer whether it actually does.

This brings me back to the topic of feedback latency which builds on the same decision tree model. How far down a branch do we go before we get meaningful feedback on the root decision?

It turns out that the practice of creating and releasing software in thin slices, getting continuous feedback from the real world about which decisions were the good decisions early and often, is also highly compatible with preserving optionality for longer.

Big Design Up-Front could be characterised as having very high feedback latency – we spend a lot more time climbing the wrong trees. It can also be characterised as closing off alternative branches too early – long before we’ve got anything real to go on – and collapsing optionality in the absence of any evidence that we’re on the right execution path.

In Extreme Programming, we delay committing to decisions – especially decisions that will be hard to change – as long as we reasonably can. This helps us to keep things as simple as we can (Y.A.G.N.I.), and it also helps to keep our options open for longer.

Another way we can keep our options open is by hiding the “how” behind some kind of abstraction. An interface, for example, describe what a module should do, but we can defer and/or substitute the choice about how it does it – the execution paths below it in the decision tree – more easily.

The outside-in test-driven design process described in Steve Freeman and Nat Pryce’s highly-recommended book Growing Object-Oriented Software Guided By Tests uses mock objects to define abstractions (contracts) before committing to any implementation.

This keeps options open for longer, and also has the benefit of allowing us to solve one problem at a time. It also provides a flex point in the architecture that keeps our options open in the future, should needs change. Checking out the basket charges the customer’s credit card. How? We’ll get to that later, but we know what the basket needs to tell the payment processor to do.

This was the original intent of mock objects – a tool for defining contracts for object interactions without committing to implementations. Mocks are a design tool, not a testing tool.

One final thought: software developers tend to be very “how”-oriented (we like solving problems), but – like the myth that pigs can’t look up – it’s also a myth that we can’t look back up a decision tree and understand the real needs that are driving the design. It’s very much in ours and our customers’ interests to do so.

The arrow of time doesn’t only run in one direction.

Feedback Latency, Learning & “Productivity”

Six teams are given a goal – to guess a random 6-digit number.

Each team gets feedback on their guesses at different stages. Team A gets feedback immediately after they guess a single digit. Team F gets feedback after guessing all 6 digits. The feedback is a simple “Right” or “Wrong”.

We can visualise their guesses as a decision tree. Team A might guess the random number starts with 7. Team B might guess it starts with 6-4. Team C might guess it starts with 2-1-0.

			
Start
│
├── Team A
│   └── 7
│
├── Team B
│   └── 6
│       └── 4
│
└── Team C
    └── 2
        └── 1
            └── 0

		

Each sequence of guessed digits represents a branch of the decision tree.

Team A learns that the first number isn’t 7. Team B learns that 6-4 aren’t the first two digits. Team C learns that 2-1-0 aren’t the first three digits.

After Team A’s first guess, there are 900,000 remaining branches that could possibly be the random 6-digit number. Uncertainty about what the number is drops by 10% – all the branches that start with 7 are eliminated.

After Team B’s guess, there are 990,000 remaining branches to choose from. Uncertainty drops by 1%. All the branches that start with 6-4 are eliminated.

After Team C’s guess, there are 999,000 remaining branches. Uncertainty drops by 0.1%. All the branches starting with 2-1-0 are eliminated.

And after Team F’s first guess, uncertainty drops by just 0.0001%.

It might take Team A as many as 60 feedback cycles to pin down the 6-digit number. It might take Team F a million. I know which team I’d bet on to guess the random number first.

This is the power of feedback latency. With latency L = 0, Team A has a massive learning advantage over Team F, but also an order of magnitude over Team B, two orders of magnitude over Team C, and so on. Team A outlearns the other teams.

Another way of thinking about it is that Team A spends a lot less time exploring the wrong branches. All 6 teams have the same odds of barking up the wrong trees – but Team A doesn’t waste time climbing them. Team F climbs every tree all the way to the top until it finds the right one.

The implication is that – if we consider learning from feedback the real source of value in software development – “productivity” can be characterised not as how much “stuff” we ship or how fast we ship it, but how fast we learn from it.

And how fast we learn from it is hugely impacted – orders of magnitude – by feedback latency. We may bark up a lot of trees, but we don’t climb them until we have credible evidence that the squirrel we’re after is at the top.

The disparity between output and feedback – Team F outputs 6x what Team A does in each cycle, but still gets 1x the feedback – creates this problem. Fans of Lean thinking may recognise it as a batch size problem. But most of the literature about the impact of batch size on system performance tends to focus on queues and delays, and ultimately lead times and throughput.

We pay lip service to learning, but don’t address it directly. Feedback latency reframes the issue to focus specifically on learning (reducing uncertainty), and its impact on value discovered* in the feedback is arguably much, much larger.

Importantly, feedback latency isn’t a measure of time before we get feedback – it’s a measure of unvalidated causality. A team working slowly can still easily outlearn teams working much faster if they have lower feedback latency.

To illustrate the impact of feedback latency on reducing uncertainty, using the random 6-digit number example, I had Claude quickly knock me up a simulation. It almost works, too 😉

* And there we were thinking value’s created in development. But what we’re really shipping is questions, and only reality has the answers.

Even If We Could Extend The Horizons of Autonomous Coding Agents, Does That Mean We Should?

Thinking about the horizon of autonomous agentic workflows. I visualise them as sequences of decisions that are dependent. If decision D1 is correct, D2 is more likely to be correct. If D1 is wrong, then D2 is more likely to be wrong, too. Errors propagate and compound down the sequence.

How many decisions can an agent make before the probability of an error-free end result drops off a cliff?

There are two factors here I’m considering:

1. The probability a decision is correct, P

2. The probability that an error will be caught before it propagates and compounds, C

The reliability of a decision in the sequence, R – the probability that a decision will be correct, or if it isn’t, the error will be caught before it propagates – would then be:

R = 1 – (1 – P)(1 – C)

If R = 0.99 (99% reliable), then the odds of an error-free result after 10 decisions – like, say, a few lines of code generated – are 90%. After 100, they’re just 37%. And after 1,000, they’re a miniscule 0.004%.

Physics predicts that LLMs are extremely unlikely to get significantly more reliable (see the research paper “The wall confronting Large Language Models”), though we can use them in ways that reduce the risk of errors (see my CRESS principles for context engineering).

So if we want to extend the horizons of our coding agents, we turn our attention to C – how strong are our quality gates, and how soon do decisions pass through them?

It’s really a testing/feedback problem. Again.

But even if we could get to R = 1 and extend the horizon to thousands or tens of thousands of autonomous decisions, the longer agents work without human feedback, the more decisions go unvalidated by real-world use. So it would, as far as learning where the real value is, be highly undesirable.

The Solution To BDUF Isn’t Faster BDUF

Big Design Up-Front – making lots of design decisions before getting real-world feedback on any of them – fails at any speed of decision-making. Indeed, the faster we make design decisions, the more we tend to fail.

Most folks make a category error of approaching design as a shopping list of decisions. We decide A, B, C, D. If it turns out B is wrong, we can just change B.

But it’s not a list – it’s a tree. Each decision constrains future choices. If B is wrong, C and D may well be wrong, too, if they’re consequences of B.

And we might also make the mistake of thinking we can just unpick the decision tree, but it’s not as simple as that. First of all, we have to distinguish the leaves from the branches. What is the root decision that we got wrong? And all we see in the resulting code is leaves.

Also, dependencies between decisions don’t operate like dependencies in code. So we’d have to extract an entire branch of decisions from an orthogonally-interconnected architecture. Decision X, Y and Z may have logical similarities that lead to shared modules that exist independently of whether features I, J and K reuse them.

BDUF is like a game of Play Your Cards Right where they don’t turn over any cards until the end. And it doesn’t matter how fast you play it, or how many goes you get, you’ll almost certainly lose.

The key factor in dramatically improving our odds of making good design decisions is how many consequent decisions we make before we get real-world feedback – how far we keep driving down that road after we make the turn. Feedback latency is a critical factor in design success.

And an AI Maserati’s just going to make things worse – we can go even further in the wrong direction, even faster.

So when someone boasts that Claude Cowork generated a PRD for 50 features in an hour, I can’t help thinking “Tell me there are no users without telling me there are no users”.

Can I back up what I’m claiming here? Yes, I think I probably can.

You’re looking in the wrong place. The value’s in the feedback, folks – not in the plan.

Return of the Revenge of the Son of Software Process Engineering

I watched an interview recently with Claude Code creator Boris Cherny where he talked about how he doesn’t prompt anymore, he just “writes loops” that do the prompting for him.

Putting aside the quality of the end product that’s producing, it’s fascinating watching a whole generation of AI-assisted and agentic developers reinvent something that’s been around for a long time.

The last time I kept my hands clean with Software Process Engineering (I could have said “got my hands dirty”, but SPE always felt like the opposite to me) was when I was Development System Architect at Symbian in 2006-2007. Ostensibly, my job was to model Symbian’s software engineering processes, so we could add that to the mountain of other process documentation teams could ignore.

You model engineering processes in pretty much the same way you’d model any business process. There are goals. There are roles. There are workflows. There’s information. There are rules. You get the picture.

I’d created my own UML profile that extended the official Software Process Engineering Metamodel – originally designed to enable teams to customise the Rational Unified Process (which was only ever intended as a template or a toolkit for defining processes, not a process in its own right). I incorporated a metamodel for modeling goals and performance measures based on the Balanced Scorecard, and some ideas about mapping goals to processes and processes to system use cases – “You do this step using Perforce” sort of thing.)

I, of course, had lost all faith in SPE by 2006. But I could do it, and do it well. I just knew that nobody touching the code was likely to ever look at these models. Because once upon a time, I was one of those coders being “programmed” by a methodologist, and I didn’t.

But the underlying conceit of SPE – that teams are factory machines that can be programmed using the metamodel – appears to be experiencing a renaissance of late with the rise of “harness engineering”. I see people defining development workflows, with goals and roles and workflows and information and rules… You get the picture.

They might not realise it, but this is software process engineering. I even see folks claiming to have codified entire “teams” of agents – each with their own goals, roles, workflows, information and rules – that interact and coordinate in wider workflows.

And it suffers from the exact same delusion at its centre – a programmable machine that executes instructions, turning use cases into realisations, turning realisations into class models, turning class models into code. Like in a factory.

In reality, the “machines” are non-deterministic and the outcomes are by no means guaranteed. The only time anyone ever did it the way I drew it is when it was me drawing and doing. My sphere of real control extends no further than me. And even then, there are times when I – quite rightly – don’t listen to me.

Any process model you define will be – at best – an abstract approximation of how you do it. And if it wasn’t built by observing what you really do, it won’t even be that. 92% of Java developers who said they did TDD actually didn’t do anything even in the ballpark when observed.

I recommend not going down this road. It’s a Fool’s Errand – whether you’re codifying development processes for teams made of people, or for teams made of hyper-scale token predictors. These folks are seriously underestimating how incredibly hard it is, and the end results show it.

I recommend that workflows be controlled by Actual Intelligence that can learn and adapt in the messy and unpredictable real world. In my “Ralph loop”, I am Ralph.

Nobody in history has ever described software development in machine-executable detail and had it actually work at scale. Sure, you might be the first. Anything’s possible.

But the fact you didn’t know how many truly great software engineers have tried, or that software process engineering was even a thing with a name (and a metamodel), doesn’t bode well.

If you’re curious, though, here’s a guide to the SPEM by Sparx Systems (who seem to have added it to their UML modeling solution since my Symbian days).

A Car Crash In Slow Motion

Since I’m among friends, I hope I can be open with you.

I started Codemanship 17 years ago in my late 30s, as a response to being asked by a recruiter for the gazillionth time “Why are you still a software developer?”

I’d been contracting for 12 years, and been programming professionally for 18, and that is what I do. I passed through lead developer roles into architect and then senior/head architect roles, and decided to walk my career back to being hands-on as a developer, but with enough authority and control over how my teams worked to do a good job – despite management.

Over the previous decade, I’d spent more and more time mentoring developers, as well as bits of structured training here and there.

The job that flipped the switch in me to make the jump permanently to that role was working as a Software Development Coach – a title I invented for myself because I didn’t like the one they’d given me (Technical Architect) – at BBC Worldwide. Even if I say so myself, I made a difference there. Not just to one developer or one team. Software development at BBC Worldwide was different by the time I moved on.

I didn’t move far, though. The lunchtime talks and workshops I’d instigated were increasingly being attended by software engineers from down the road at BBC corporate. After the Worldwide gig, I spent several years popping in to various BBC sites in London and Manchester running training and a very successful peer-led coaching experiment in TV Platforms (the iPlayer folks).

And in between, I ran my last team full-time for a small new consulting company owned by two guys who knew nothing about software development. I won’t go into the details, but that last contract left such a sour taste – and the BBC work was looming – that I finally did it and became a full-time trainer and coach, and founded Codemanship in 2009.

If you’ve started your own business, you’ll know that the first 2-3 years can be tough. I had savings, but one client was never going to be enough. I had a pretty modest goal: make half what I was making as a contractor, doing something I really enjoy. (And I like to think it has real value, too – we’ll circle back to that.)

So I found a small building on Blackfriars Road in South London – opposite Southwark tube station and, importantly, a decent boozer – and rented training rooms for weekend courses for folks funding their own career growth. I didn’t have the business contacts, but I knew a lot of software developers.

For a fraction of the price of corporate training, groups of ~16 enthusiastic folks gave up their weekends and a few hundred quid of their hard-earned cash to do what you might recognise as the ancestors of the Code Craft courses that I’ve run for many clients and for thousands of developers since then.

It was such good value that folks flew in from as far afield as Russia – back when they could – and the US and Canada. They’d book a hotel for a day or two after to see the sights – make a city break of it.

We’d spend a day TDD-ing and refactoring and SOLID-ing and wotnot, and then retire to The Ring to reflect on the day and talk around what we’d covered, going into really cool asides and general chit-chat.

Training has never felt like work to me – hence my ambition to do it for my living – but these weekend workshops felt even less like work, and more like tech meetups or conferences. Y’know – the interesting bits in between and after the talks. If there were 17 of us in the workshop, there’d usually be 8-10 of us in the pub after. That’s a sociable ratio.

Private workshops for corporate clients rarely end like that. (With exceptions, of course – Hi to the folks at Hostelworld in Porto and their magically replenishing beer fridge!)

The weekend workshops felt much more like community events than corporate training. They’re very hands-on, folks are pairing up and meeting new people, and I am “Jason from Twitter/LinkedIn/Jason’s Blog”, and not just “That Guy Who’s Running The Course I’ve Been Told To Go On”.

Although it never really occurred to me, these original training cohorts became my busy bees, buzzing from gig to gig, gaining seniority and influence year on year, until corporate orders started coming in from places they were working.

Codemanship’s client base grew quite organically from things like this, as well as from my activities within the developer community – speaking at events, organising conferences, going to meetups etc.

In fits and starts, the business grew. Sure, there were fallow periods, and there were busy periods. And I didn’t feel the need after a while to run out-of-hours training. Organising small, low-priced public workshops is a lot more work per £ it brings in. There was just enough corporate training and coaching to keep the lights on. And, generally, the trend was “number go up” by roughly 10% a year.

By 2019, I was on track to achieve my goal – half my contract income doing something I love. At the time, I really didn’t think I was reaching for the Moon.

Then, in early 2020… Well, you know what happened in early 2020.

Business just disappeared for 6 months. But then something fell into my lap, courtesy of Nat Pryce, that ultimately led to autumn 2020 to autumn 2022 being the best two years the business ever had. And although I knew an ongoing coaching gig was going to be financial aberration, and that I shouldn’t get used to it, during the same time, the training side of the business grew too – beyond my original goal.

The summer of 2022 was the peak. I was better off than I’d been for many years, and was even about to put in an offer on a detached house in Wiltshire – with a garage and a garden and a utility room! Imagine that -having a room just for utility! Folks like me living in an expensive-ish London postcode can only dream of such luxuries.

Then, as with all peaks, it’s downhill on the other side. Unwittingly, I’d been the beneficiary of a hiring frenzy bankrolled by free credit during the lockdown era. It was only then I realised just how closely my sales tracked with entry-level hiring. In the room, I saw plenty of senior developers. But, it turns out, those workshops wouldn’t have been booked at all if it wasn’t for the junior intake in the room. I’d become the Onboarding Guy.

In 2023, sales dropped 65% – an almost exact match to entry-level hiring here in the UK and Europe. And the ongoing coaching gig had already ended with the rapid rise in interest rates, so that generous tap was turned off.

That’s a big pay cut.

But, I’d been through these cycles before, and had weathered them with savings and loans. So that’s what I did. I didn’t panic. I didn’t think “Shit, I need to get a contract”. I thought “This, too, shall pass.”

In 2024, it didn’t pass. Entry-level hiring fell further. Layoffs, layoffs, layoffs in the news. But still, I didn’t panic. I had savings. I had time.

In 2025, hiring started to recover, but not entry-level hiring. (Hey, it’s a good job all those senior developers know TDD, right?)

But by the middle of the year, something gave me hope and made me stay my course. By this time, after two years of experimenting with and researching AI-assisted coding, I’d figured something out – the principles and practices that I’d been teaching for 25 years, far from becoming less important, were becoming more important than ever with the rise of AI.

Clever Jason! They’ll be queuing outside my door any moment.

As 2025 went on, and more and more good data rolled in, that position just got more and more solid.

Any minute now…

AI coding tool adoption passed a tipping point over the Christmas break, as many engineering leaders finally found some time to play with the technology, got it to build them a Calendar app or a TO-DO list in a nanosecond, and came back to the office and proclaimed to their teams “You will use this!” Because real software development is exactly the same as doing self-contained mini projects by yourself for fun.

So we’ve been seeing more and more teams finding out what I – and many others – figured out a year or more ago. Without solid engineering foundations, that stuff will hurt you. It’ll slow down your release cycles, it’ll make your lead times longer, and it’ll create a growing mountain of quality problems that leaks into production. The evidence is now overwhelming that’s really what’s happening for the majority of teams.

And when I’ve spoken to engineering leaders and polled them about it, they agree that engineering foundations are more important then ever.

Any minute now…

In the meantime, my savings are gone, my credit cards are maxxed out, and orders in 2026 are down to 10% of what they were five years ago.

Two months ago, I pivoted back to where I started – out-of-hours training for people funding their own learning. If your boss doesn’t see the value in engineering foundations, maybe you do. And, if I set the price accordingly, maybe I can put that kind of training within your reach.

And this actually started well. The first few workshops on Tuesday evenings and Saturday mornings sold out. And they’ve really taken me back to that training room on Blackfriars Road, because they’ve felt much more like community events with some training thrown in to give us something to talk about.

I’ve really been enjoying the after-workshop discussions, and the ratio has again been very sociable – typically more than 70% stay on to chat. And this gave me hope.

To quote John Cleese in the 80s movie Clockwise, “It’s not the despair, Laura. I can stand the despair. It’s the hope.”

After those first few, interest has dropped off dramatically. I suspect the 60 or so folks who’ve bought tickets are the extent of the market within my reach.

I do not know where it goes from here.

So, after a little cry early this morning – not kidding – I think maybe it’s time to do some adulting and let go of this particular life goal. I can’t hold out any longer. In fact, I should have let go last year because now I’m a year older and in a real hole.

I don’t know what I’m going to do next. I find myself 55 years old and having not been employed by anybody else for 17 years. Friends will know that I’ve stayed very hands-on and current throughout that, and am still very capable of working as a developer and also leading teams. And – having used so many over the years – I can learn programming languages, tools and tech stacks very fast, even at my age.

But it’s not you I’d need to convince. As I understand it, job applicants have to contend with so many layers of corporate gatekeepers these days (human and AI) – who wouldn’t know a software developer from a hole in the ground – that I suspect I will struggle to get in front of the right people.

The final public workshops will go ahead as planned. Folks have bought tickets. And there are still some places left – I’d be very happy if you could join us. This could well be your last chance to experience what I’ve spent 17 years making a unique, hands-on training experience.

June 16 & 20 – Refactoring

June 30 – Specification By Example

July 7-9 – Code Craft (one final public voyage for my flagship 3-day workshop)

And if you ask me to run a private workshop for your team, I’m not going to say no. I’d be a fool to.

But I’m officially now “between careers”. Where that ends up at my age… I guess I’m going to find out.

Codemanship has turned out to be half my entire career. I’d hoped one day it would be my retirement. I love to do this job when I’m given the chance. And, if you follow me on social media, you probably know I do it even when nobody’s paying, which has been most of the time. (And, of course, there were times when I didn’t realise I wasn’t being paid – but that’s the life of a small business owner.)

I can’t complain. It’s been my dream job, and I’m very grateful to everyone I’ve met along the way.

Feedback With A Face

One handy thing about living in London, if you enjoy stand-up comedy, is that so many comedians test new material here in small venues – often playing “works in progress” to audiences of just a few dozen.

Stewart Lee famously iterates his show over many, many performances at the Leicester Square Theatre before he takes it on tour to bigger venues and has it recorded for TV and DVD.

Here’s the thing: comedy requires feedback. Immediate feedback. Not an aggregate report at the end of the show, but in-the-moment feedback about how a joke’s landing. In big theatres, audiences can become that faceless aggregate, but in 100-seater venues, every data point has a face.

And that matters. It matters when you can see the faces and hear the responses from your audience. Because now each one of them matters, and that’s a very different kind of feedback to being told that “27% thought that the routine about Prince Andrew went on a bit too long” after they’ve all gone home.

I hear developers all the time complaining that there are just too many users to get that kind of feedback-with-a-face. I say that’s a choice – like skipping the warm-up gigs at Old Rope at the Comedy Store and taking your show straight to the O2 Arena.

It’s worth cultivating small audiences to test new material on. Sure, you don’t get to see the aggregate trends – only big audiences can give you that. But you can see their faces, and you know immediately if the joke’s aren’t landing. And if you’re going to die on stage, it’s preferable not to do it in front of 20,000 paying punters.

One final thought: I’ve observed our industry morph from one where the data points had faces and individual users’ experiences mattered to one where we only play the proverbial stadiums, and we only see the trends, not the faces.

This, I suspect – while not a direct cause – has been an enabler of “enshittification”. It’s much easier to do that to a faceless aggregate.

Faster Feedback -> Better Outcomes

The impact of feedback loops like testing in software development can be as profound as it is widely misunderstood.

Movie-making had a similar problem up until the 1960s. Crew shoots a take during the day. Director has to wait until the film’s processed so they can watch “the dailies” to check for any mistakes nobody noticed at the time – like an extra using an iPhone in what’s supposed to be 1889 – and to see if the shot actually works dramatically, comedically etc.

If they wanted to fix it, back in the day, that could mean rebuilding the set, or transporting everyone – cast, crew, equipment, costumes, props etc – back to the location. Remounting shots is a big deal.

In 1960, comic actor and director Jerry Lewis started using “video-assist” while working on The Bellboy. Takes were captured simultaneously on film and on video, so the director can check each shot in “video village” immediately after the take. If a joke’s not working, they can see straight away and adjust for the next take. By the mid-60s, the technology had been refined using a beam splitter to ensure the video captured was showing exactly what the film camera was recording. WYSIWYG.

It made a big difference. When we move the feedback much closer to the action and the myriad decisions made in just a single shot, fixing problems gets much quicker and much, much cheaper. So – unsurprisingly – more problems get fixed.

Cinephiles like myself may have noticed a tangible leap in the quality of films being made during the 1960s and early 1970s, as this technology became mainstream.

In software development, we have our equivalents of “video-assist” – techniques we can use to bring the feedback much closer to the decision, making mistakes much quicker and cheaper to fix.

A good example is developer testing. Instead of making a whole bunch of changes to the code and then testing all of them, we make one change and immediately run to our equivalent of “video village” – a unit test suite, for example – to check for problems.

Teams that rely on downstream testing are doing the equivalent of waiting to see the dailies. When problems are caught, fixing them becomes a bigger deal. Likely as not, the developers have moved on. The set’s been struck, so to speak, and remounting those shots is a bigger deal.

What other examples can you think of where we move feedback closer to the decision in software development?