Chance favors the (theoretically) prepared mind

tl;dr
It’s a common refrain that theory follows practice, almost like a “post-mortem”, leading many to question theory’s value, especially in our modern, data-heavy world. Why invest in theory? I think this mindset derives from an overly narrow view of the causal chain of discovery. If you zoom out, you can see the myriad ways that theory is the engine that drives tinkerers to new discoveries. I argue that we need to preserve a place for theory in our modern world, lest we lose sight of some lessons about how science and society progress.

Theory is in a bad place these days. Following the triumphs of the 20th century, as we have moved towards studying complex systems that are perhaps now beginning to yield their secrets to machine learning, I think it’s fashionable to ask why we bother with theory at all—let’s just collect all the data and let some GPUs tell us what it all means. This mindset is not new to the AI era, though. Versions of the argument that theory is of limited value because it often comes after engineers have made all the practical advances have been around as long as I can remember. Basically, the theory is like a “post-mortem” to explain how things work to a couple of eggheads long after its usefulness has been established.

For instance:

I think these arguments arise from an overly narrow view of progress. The problem is that the timescales of the applications of theory are, in fact, so long that we get the assignment of cause and effect mixed up. Let’s take the example cited above of circuits and Maxwell’s equations, the equations that govern electrodynamics. Yes, circuits certainly predate Maxwell’s equations, and so if you look at it in that way, sure, it’s a “post-mortem’.

Let’s zoom out a bit, though. Did people just randomly put pieces of metal together and find that they formed circuits? Not at all! At the time, the idea (theory, if you will) that electricity was a fluid (Ben Franklin) that could move from one place to another served as the basis for circuit design. I’m not sure, but I would assume that theory served as the basis for circuits.

We can do the same exercise on the other side. Take the invention of the radio by Marconi. Was his invention just the result of random tinkering? Not at all. His work already heavily relied on the wave theory of electromagnetic radiation (confirmed by Hertz), without which there would be simply no way for him to make any progress whatsoever. I can assume that these theories were well-established, probably to the point of being taken for granted.

Of course, one might argue that in life sciences, we rely a lot more on experimentation and serendipity, so the relevance of theory is lower. I think there is a sense that we should therefore be doing a lot more experimentation. See, for instance, the following tweet from Rux Teslo, made in reference to the above tweet about theory lagging practice:

I am certainly sympathetic to this point, and I would agree with Teslo that we need a lot more experimentation. And certainly serendipity is often brought up in the context of drug development. But here’s the thing: the space of all possible experiments is impossibly large, and theory serves as a (sometimes invisible) guide through this space.

Let’s take a look at penicillin, a seemingly classic case of serendipity: Fleming leaves a Petri dish out, which becomes moldy, and the mold kills the bacteria. From there, penicillin is derived, and a new era of medicine is born, seemingly by chance, irrespective of the particular details (“mechanism of action”) by which penicillin’s effects are mediated. But even here, the pattern is actually the same. Zoom out a bit, and the very basis of this discovery is the germ theory of disease, formed some 60 years earlier by Pasteur. Without germ theory, there would be no basis for this observation to have any meaning whatsoever. Zoom out in the other direction as well: discovering the genetic basis for penicillin resistance is critical for the molecular cloning that fueled the field of biotech.

Same for cancer chemotherapies. Cisplatin was discovered by noticing that an electrode had the effect of stopping bacteria from dividing, so the reasoning was that it might have an effect on cell division in cancer. However, this whole chain relies on the very knowledge that cancer is a disease of our own cells dividing uncontrollably. Indeed, for much of human history, it was thought that cancer was actually a disease caused by foreign objects or internal imbalances of bodily fluids. The conceptual innovation was required for someone to make the connections required to realize the significance of the observation.

Anyway, again, none of this is to say that serendipity doesn’t play any role, nor that we should have fewer rather than more clinical trials (I would certainly argue the opposite). But I do think that amidst all the excitement around high-throughput data gathering, machine learning, and the like, we should be careful not to underestimate the value of theory. We may not see it immediately, or even in the short term, but we ignore theory at our peril. It’s what prepares our mind to turn change into serendipity.

PS:
Also, it is notable that all these discoveries were made by people who were deeply immersed in their disciplines. These were not random people doing random things. These were very much people with prepared minds. There is a strain of anti-establishment sentiment that says that institutions of learning are holding back knowledge and progress. I think the evidence simply does not support that view.