The Difference That Doing Makes

By 1987, an estimated sixty to ninety billion dollars of institutional assets was covered by portfolio insurance, a strategy built by Hayne Leland and Mark Rubinstein on the option-pricing mathematics of the 1970s. It aimed to give a portfolio the downside protection of a put option, without buying one, by selling stock index futures as prices fell and buying them back as they rose. In the models, it worked.

The protection did not rest on a pattern in past data; it rested on assumptions that held only while prices moved smoothly, trading stayed cheap, and the fund was small enough to trade without moving prices. Putting real portfolios behind it meant relying on those assumptions in a different setting: a crowded market where many funds ran the same rule at once, and the selling itself moved prices and dried up liquidity. One fund running the rule and many funds running it are not the same intervention.

On 19 October 1987 that is exactly what happened. As prices fell, the models called for heavy selling; that selling deepened the decline, and lower prices called for still more. Official investigations treated that procyclical selling as an important amplifier of the crash, though its contribution remains disputed. All three assumptions failed together, and the protection proved much weaker and less reliable than clients had expected.

What kind of problem is this?

When the question you can answer and the question you need answered sit at different levels, and ordinary language lets you slide between them without noticing, that is a specific shape of problem. One is about observation: what goes with what, the way this signal accompanied that return. The other is about action: what happens when we deploy capital, and what would have happened had we chosen otherwise. And the discipline that has spent the last three decades making those levels precise is causal inference.

So let’s borrow.

This is the multi-model move: recognise the shape of a problem, find the discipline that has thought rigorously about that shape, and import its frameworks deliberately rather than reinventing them from scratch.

The framework comes from Judea Pearl, the computer scientist whose work on probabilistic and causal reasoning reshaped artificial intelligence and was recognised with the 2011 Turing Award.

Pearl sorts causal questions into three levels, or rungs on a ladder. The first rung is association, or seeing. It asks: what does observing X tell me about Y? This is the rung of correlation, prediction, and curve-fitting, where much of predictive statistics and standard machine learning operates. The data here are, in Pearl’s phrase, profoundly dumb about causes.

The second rung is intervention, or doing. It asks: what happens to Y if I set X by deliberate action? Pearl gives this its own notation, the do-operator, because doing X is not the same as seeing X. His example is the barometer: the needle and the storm move together, but turning the needle by hand does nothing to the weather, because the correlation runs through atmospheric pressure your hand never touches. To answer a doing-question you need causal assumptions or an experiment, not observation alone.

The third rung is the counterfactual, or imagining. It asks: given what happened, what would have happened had I acted otherwise? This is the most demanding rung, of regret and attribution; answering it requires a structural causal model, or structural assumptions rich enough to identify the counterfactual. “Would the patient have recovered had we withheld the drug, given that she received it and recovered?” is a harder question than “does the drug work on average?”

Pearl’s formal point, sharpened by Elias Bareinboim and colleagues in the Causal Hierarchy Theorem, is that the ladder is a ladder, not a ramp: in general, no summary of lower-rung data settles a higher-rung claim on its own. To climb, you must add causal assumptions.

Backtests do not have a rung; the claims made from them do. A regression of returns on a signal is associational. A historical replay is a model-based policy simulation, counterfactual only when it asks what this same realised history would have looked like under a rule we did not run. Either way, replaying fixed historical prices does not by itself identify what deploying the rule will cause. The allocation question is interventional: what distribution of outcomes follows if we deploy this policy at a given size, cost, and market state?

Absent experimental or live-deployment evidence, we answer that question with rung-one evidence plus causal assumptions: that the observed relationship is policy-relevant, that the mechanism or latent state it tracks remains stable, and that the relationship survives our acting on it.

A confounder is the familiar worry: a signal predicts returns, but a third, unobserved factor, such as a shared funding condition, drives both. In causal inference that is fatal to the naive inference, though sometimes identifiable by other means. For an investor it need not be fatal: conditioning on the signal still selects exposure to the latent state that drives returns. The signal is a dial, not a lever, useful while it stays informative, not a cause you operate.

Counterfactual attribution and alternative-history post-mortems belong on rung three. Arithmetic attribution is an accounting decomposition, not a causal query, and belongs on no rung. “What would our worst year have looked like had we cut duration that January?” is a counterfactual about a single realised path; the alternative never ran.

Textbook medical examples treat the mechanism as stable enough that a trial reveals it rather than changing it. Markets are less forgiving: the mechanism that generates returns often includes beliefs about that mechanism. Intervening on a trading rule need not freeze the rest of a structural model; prices, beliefs, liquidity, and participation may all respond downstream without any mechanism breaking.

The harder point is that a trade by one small fund and the same rule run by many large funds are different interventions, and the second may run under different market mechanisms. So the intervention must be specified precisely: size, timing, execution, publicity, and how many others run it, with the feedback it sets off modelled as a consequence, not folded into the action. Markets do not suspend the Pearlian lesson; they make it harder, because the stability an effect needs to carry across settings is the first thing trading at scale disturbs. The real question is whether an effect found at small scale survives large-scale deployment at all.

This is the same reflexivity the series has met before. It is the strategy decay of Why Good Strategies Stop Working, where a rule’s fit erodes as its environment adapts, and the active inference of The Model That Fights Back, where the system updates on your prediction. Goodhart’s law names one way this happens. Charles Goodhart, writing about monetary policy in 1975, observed that any statistical regularity tends to collapse once it is used as a target for control. The ladder does not imply this on its own: seeing and doing would differ even if a pattern survived being acted on. In practice patterns often do not survive, and Goodhart names one common reason. Once people know you are trading on a rule, they adjust to it, and the pattern you measured changes.

The first move is to label evidence by its rung. When someone shows you a backtest, ask whether it claims the pattern held in the past, which is rung one, or that it will hold when you trade on it, which is rung two. The presentation usually blurs the two, and separating them is most of the work.

The second is to state the causal assumptions that take you from rung one to rung two. Those assumptions carry the decision. Pearl’s other tool is the causal diagram: which variables you think cause which, and which shared causes you allow for or rule out. At a minimum, write down what you think the signal is tracking, whether it causes returns or shares a hidden driver, and what must hold for the strategy to keep working once you trade on it. In a market that reacts to you, the diagram needs to include time: this period’s trade affects next period’s prices and beliefs, which a static diagram leaves out.

The third is to treat counterfactual attribution as what it is: a claim about a history that did not happen. Rather than “cutting duration would have saved four hundred basis points,” say “on our model of how the positions interact, it would have saved roughly that much.”

The fourth follows from the same point: because acting is itself an intervention, capacity, size and decay are not details to add later; they are part of specifying the intervention, and of whether an effect found in one setting transports to another. An effect measured while trading small need not survive market-moving deployment, even though both remain rung-two questions, because trading at that scale changes the system you measured it in.

When does this lens apply? Whenever you reach from an association to a claim about what acting will do, or would have done. That is most of investing, and the shape recurs far beyond it. A doctor infers that a drug speeds recovery because treated patients recovered faster, when the milder cases may have been the ones treated. A restaurateur adds live music because the busiest nights have it, when weekends bring both the crowds and the band. A club credits a new coach with its turnaround because results improved soon after the appointment, when a bad run tends to end on its own. Each reaches from what accompanied what to what an action will bring about, and the association never settles the action on its own.

Knowing the ladder does not lift you off it. What it does is more modest: it tells you when you are climbing from one rung to the next, so you can be honest about the assumptions carrying you up and size your conviction to them, not the bare correlation. And this applies to you as much as to anyone: noticing that acting changes the system does not exempt your acting from changing it.

The discipline is in asking: is this evidence about what I have seen, or a claim about what my doing will cause, and what must be true of the world for the one to license the other?

References & Further Reading

1. The Book of Why: The New Science of Cause and Effect: Judea Pearl & Dana Mackenzie

2. Causality: Models, Reasoning, and Inference: Judea Pearl

3. Causal Inference in Statistics: A Primer: Judea Pearl, Madelyn Glymour & Nicholas P. Jewell

4. The Seven Tools of Causal Inference, with Reflections on Machine Learning: Judea Pearl

5. On Pearl’s Hierarchy and the Foundations of Causal Inference: Elias Bareinboim, Juan D. Correa, Duligur Ibeling & Thomas Icard

6. Problems of Monetary Management: The U.K. Experience: Charles Goodhart