Striking evidence for forced experimentation

February the 5th, 2014 was a typical 7 - 10 degrees Celsius in London, and rainy. That morning, as they did every weekday morning, millions of commuters woke up and got ready to travel to their places of work in the UK capital. Normally many of them would use the London Underground and its network of 270 tube stations, but this morning, 5th of February, 171 of those stations were closed due to a strike. Some commuters could take their normal route, as they did every morning, but many would have to find an alternative route to work.

Habit is one of the great predictors of human behaviour (along with what the other people around you are doing). For a lot of what we do, we rely on repeating what we’ve done before. In learning theory this is ‘exploitation’ — taking advantage of what you already know, and what has already worked for you before. Its counterpart is ‘exploration’ — trying something new, with an uncertain outcome. With exploration you learn something new, even if what you learn is that whatever you tried wasn’t worth it.

Exploration-exploitation is one of the fundamental trade-offs. There’s no right answer. It depends on what you have to lose, how much you think you know about your options, and how complex the choices you face are.

For commuters, the choice is a very complex one. Travel routes are famously hard to solve. They belong to a class of mathematical problems where the best answer is hard to find, but easy to verify. If you identify a new route it is easy to show that it is shorter than your current best option, but before you identify it, you don’t even know if a better option exists and there’s no easy way of knowing.

As well as being theoretically intractable, commuting in London also has multiple possible modes (road, rail, tube and river), which provide multiple routes within and combining modes, and each potential route has a characteristic travel time, as well as variability (one route might be quicker on average but be less predictable), along with idiosyncrasies which might make different routes attractive (like passing a great coffee shop). All this complexity makes the best route for each individual commuter inscrutable. The only way to solve the problem is to try different routes out, and evidence from the strike suggests that commuters under-explore. They rely too much on habit, to their own long-term cost.

In their paper, The Benefits of Forced Experimentation: Striking Evidence from the London Underground Network, Larcom et al (2017) report analysis of London Underground commutes, gathered from 3.5 million travelcard (’Oyster’) IDs. From these IDs they identified regular commuters who made the same journey, entering and exiting via the same stations, every working day between 7am and 10am from January 15th 2014 to February 4th (the day before the strike).

These regular commuters make up the population for a natural experiment. On the two strike days some get to travel their normal route (because the strike doesn’t affect the stations they use), and some are forced by the strike to experiment, trying out new routes. The focus of the analysis by Larcom et al (2017) is whether those who have to experiment keep using a new route in the days after the strike. If they switch to a new route, despite the end of the strike meaning their old route was available again, it suggests that they’ve found a new route which, for whatever reason, they prefer.

The analysis suggests a small but robust effect: those who had to experiment with a new route were about 5% more likely to switch to new route post-strike, compared to those who could rely on their established commute. That means the strike induced a behaviour change in the habits of about 1 in 20 people it affected.

Above is my recreation1 of a figure from the paper. After some fairly rigorous exclusion criteria, the researchers identified 12,346 regular commuters who experienced disruption on their normal route, and tried a different route because of it, (the “treatment” group), and 4,996 regular commuters who didn’t experience strike disruption. The plot shows the percentage following their typical route on each day. The impact of the strike (yellow bar days) is clear: people whose routes were disrupted were unable to make their normal commute. And the effect of this forced disruption can be seen in the shift in the line for the treatment-group only post-strike: approx 5% permanently abandoned their old route (the red line is lower down on the right).

Follow up analysis confirms that people found better routes. The average travel time for those who found a new route was 400 seconds quicker (i.e. 6 minutes 40 seconds saved typically). Over the years that saved time adds up. In fact, Larcom et al (2017) calculate that the strike generated a net saving of time. Despite the disruption on the strike days, the benefit for those who found better routes added up to a total greater gain than the time lost to everyone else due to the strike!

The researchers also found evidence that the benefits of forced experimentation were greater for those commuters who were likely to have a harder time finding good routes.

The London Underground map is a design classic because it famously sacrifices spatial accuracy for route clarity. Where stations are on the tube map doesn’t match where they are in space. You can see this by comparing the lines on the normal tube map that commuters use (bottom) with the same route drawn on the non-spatially distorted tube map (top):

The top map makes planning travel harder, the bottom map is misleading about the distance between stations. In fact, Guo (2011) reports that the correlation between the map-distance and actual-distance between stations is only 0.22 (where 0 is “no correlation” and 1 is “perfect correlation”).

Perhaps the most famous example of the kind of distortion introduced into people’s mental models by the official tube map is the journey between Covent Garden and Leicester Square. On the tube map the stations are a standard distance apart, but in reality they are only 260 metres from each other. As Larcom et al (2017) note “the 20-second Tube ride remains in high demand”.

Larcom and colleagues show that the strike’s effect were largest were the map was most distorting. The probability the strike permanently altered a commuter’s choice of route was higher for commuters whose journey included a part of London where the differences between the tube map and spatial reality were highest. Another kind of commuter that was particularly likely to benefit was those who travelled on slow tube lines. The map shows the number of stops, but not the differences in speed of different underground lines, meaning that commuters choosing routes based on the map had more potential to discover a surprising upside when they were forced into experimentation by the strike.

The researchers quote a BBC article which gives some examples of positive exploration generated by the strike. One commuter who took the Thames Clipper water bus said “It has been fine, the boat is a great journey. I think I will get the boat back too.”. Another who said “the walk from Liverpool Street was a refreshing change from the horrors of the Circle Line. I suspect I may permanently switch so I can cut out this, the most stressful part of my journey.”

§

The analysis is a really nice example of a natural experiment, and carries some general lessons. As the researchers summarise: “These results highlight the importance of implementing occasional routine breaks.”

We all have a tendency to over-rely on our habits, leaving potential rewards on the table, whether they are commuting routes we don’t know about, types of food we haven’t tried or people we haven’t met. Exploring is effortful, and doesn’t guarantee success, but results like these suggest that we underweight its value and in many areas of life being forced to try something new could produce a net benefit.

This newsletter is free for everyone to read and always will be. If you can afford it, feel free to chip in to help me keep writing for everyone: upgrade to a paid subscription (more on why here).

Upgrade to paid

See below for references, and other things I’ve noticed.

Larcom, S., Rauch, F., & Willems, T. (2017). The benefits of forced experimentation: Striking evidence from the London underground network. Quarterly Journal of Economics, 132(4), 2019–2055. doi:10.1093/qje/qjx020

• Analysis replication: https://codeberg.org/mephi1808/london-strike-replication the original authors share their data (kudos to them), and I asked my agent to replicate the results (which it could) and this allowed redrawing of the plot from the paper (Figure 5) shown above. Normally I would only do this for a result I suspected was wrong, wanted to extend or cared a lot about. The advent of AI means this could be done in less time than it took to write this newsletter (indeed, it happened while I wrote, with me breaking off to answer a few questions and making a few tweaks to the presentation at the end). It is the same as the result in the paper, so unlike many AI outputs, it doesn’t need the same kind of verification (flipside: if the paper is wrong, we’ve just shown that we can recreate the error). All details at the link

Guo, Z. (2011). Mind the map! The impact of transit maps on path choice in public transit. Transportation Research Part A: Policy and Practice, 45(7), 625-639. https://doi.org/10.1016/j.tra.2011.04.001

More on the Tube Map and design : The Real Underground

Previously by me: When less (communication) is more (collective intelligence): reporting on experiments which used route planning as a complex problem to test theories of collective intelligence.

Other things…

In an absolutely starling admission, the X Head of Product tweeted about an experiment in which X removed the most engaging accounts from people’s feeds. This showed that this change caused people to visit the site more often and spend longer on it when they did, reports Craig Silverman for Indicator. The accounts to be removed were the top 3% according to X’s monetisation programme, which pays accounts which generate engaging content — and make no mistake, ‘engaging’ in this context means literally ‘content people engage with’, which includes engaging with it by becoming infuriated and posting angry responses. When these conflict entrepreneurs are removed from users’ feeds, users respond by spending more time on X.

The X head of product flatly summarised “The Top 30 accounts do not post content that benefits X”.

It’s a great example of a perverse metric, where over-optimising for a narrowly defined measure (engagement generated by individual accounts) has degraded the overall experience and so made the product worse. Derek Thompson (on X) puts it like this “the best attention-maxxers are basically engagement parasites that gobble up views and, in the process, annoy the shit out of people’.

Indicator: X tries to undo its monetization mess

In the MIT Press Open Encyclopedia of Cognitive Science, Ulrike Hahn provides an excellent short introduction to research on rationality, and in particular the different frameworks (logic, probability, game theory) which contribute to our accretive, incomplete, normative account of what it means to be rational.

Hahn, U. (2026). Human Rationality in Research. In M. C. Frank & A. Majid (Eds.), Open Encyclopedia of Cognitive Science. MIT Press. https://doi.org/10.21428/e2759450.84dbfd75

“An interactive answer machine that plays a series of uncanny voicemails generated by randomised predictive phone message texts.”

From Geraint Edwards

From Lynda Barry (via @Natasha_Jay)

END

Comments? Feedback? Force me to experiment with something? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

AI declaration: I write all the words and think all the thoughts myself. I asked Gemini to check for spelling and grammar. For this post I also asked my custom agent (“Mephi”), which is openclaw, Claude and some local models as well as a bunch of bespoke preferences and protocols, to replicate the analysis in the paper (and shown in the graph).