Six teams are given a goal – to guess a random 6-digit number.
Each team gets feedback on their guesses at different stages. Team A gets feedback immediately after they guess a single digit. Team F gets feedback after guessing all 6 digits. The feedback is a simple “Right” or “Wrong”.
We can visualise their guesses as a decision tree. Team A might guess the random number starts with 7. Team B might guess it starts with 6-4. Team C might guess it starts with 2-1-0.
Start│├── Team A│ └── 7│├── Team B│ └── 6│ └── 4│└── Team C └── 2 └── 1 └── 0
Each sequence of guessed digits represents a branch of the decision tree.
Team A learns that the first number isn’t 7. Team B learns that 6-4 aren’t the first two digits. Team C learns that 2-1-0 aren’t the first three digits.
After Team A’s first guess, there are 900,000 remaining branches that could possibly be the random 6-digit number. Uncertainty about what the number is drops by 10% – all the branches that start with 7 are eliminated.
After Team B’s guess, there are 990,000 remaining branches to choose from. Uncertainty drops by 1%. All the branches that start with 6-4 are eliminated.
After Team C’s guess, there are 999,000 remaining branches. Uncertainty drops by 0.1%. All the branches starting with 2-1-0 are eliminated.
And after Team F’s first guess, uncertainty drops by just 0.0001%.
It might take Team A as many as 60 feedback cycles to pin down the 6-digit number. It might take Team F a million. I know which team I’d bet on to guess the random number first.
This is the power of feedback latency. With latency L = 0, Team A has a massive learning advantage over Team F, but also an order of magnitude over Team B, two orders of magnitude over Team C, and so on. Team A outlearns the other teams.
Another way of thinking about it is that Team A spends a lot less time exploring the wrong branches. All 6 teams have the same odds of barking up the wrong trees – but Team A doesn’t waste time climbing them. Team F climbs every tree all the way to the top until it finds the right one.
The implication is that – if we consider learning from feedback the real source of value in software development – “productivity” can be characterised not as how much “stuff” we ship or how fast we ship it, but how fast we learn from it.
And how fast we learn from it is hugely impacted – orders of magnitude – by feedback latency. We may bark up a lot of trees, but we don’t climb them until we have credible evidence that the squirrel we’re after is at the top.
The disparity between output and feedback – Team F outputs 6x what Team A does in each cycle, but still gets 1x the feedback – creates this problem. Fans of Lean thinking may recognise it as a batch size problem. But most of the literature about the impact of batch size on system performance tends to focus on queues and delays, and ultimately lead times and throughput.
We pay lip service to learning, but don’t address it directly. Feedback latency reframes the issue to focus specifically on learning (reducing uncertainty), and its impact on value discovered* in the feedback is arguably much, much larger.
Importantly, feedback latency isn’t a measure of time before we get feedback – it’s a measure of unvalidated causality. A team working slowly can still easily outlearn teams working much faster if they have lower feedback latency.
To illustrate the impact of feedback latency on reducing uncertainty, using the random 6-digit number example, I had Claude quickly knock me up a simulation. It almost works, too 😉
* And there we were thinking value’s created in development. But what we’re really shipping is questions, and only reality has the answers.
Codemanship's Blog 