Counterfactual Thinking

A sales team launches a new pricing strategy.

Sales increase.

A new training program is introduced.

Productivity improves.

A marketing campaign goes live.

Conversions rise.

In each case, the conclusion seems obvious:

The intervention caused the improvement.

But there is a problem. You observed what happened after the decision.

You didn’t observe what would have happened without it.

That missing scenario is where some of the hardest problems in business analysis begin.

A foundational concept in causal inference is counterfactual thinking.

The counterfactual asks:

What would have happened to the outcome if the intervention had not occurred?

This sounds simple.

It isn’t.

For any individual customer, employee, business unit or market, we can observe either:

What happened with the intervention

or

What would have happened without the intervention.

We cannot observe both for the same unit at the same time.

That is the fundamental challenge of causal inference.

Business dashboards are very good at answering:

“What happened?”

They are much less capable of answering:

“What happened because of what we did?”

That distinction matters because outcomes are rarely produced by a single variable.

Suppose sales increase after a new marketing campaign.

At the same time:

• competitors raise their prices

• seasonality increases demand

• the sales team expands

• the economy improves

The observed increase in sales contains the effect of all of these forces.

The causal question is:

How different would the outcome have been in the absence of the intervention?

This is also why aggregate data can be dangerous.

In Issue #019, Simpson’s Paradox, we explored how the overall pattern in a dataset can conceal what is happening within its underlying segments.

Counterfactual thinking takes the question one step further:

Even when the pattern is real, what actually caused it?

When evaluating whether a decision actually caused an outcome, use five steps.

What exactly did we change?

Avoid vague explanations such as “the strategy worked.”

Specify the intervention.

What changed that we are trying to explain?

Revenue? Retention? Productivity? Conversion? Cost?

What else could have produced the change?

This is where confounding variables and selection effects become important.

What would probably have happened without the intervention?

Possible approaches include:

• randomized experiments

• control groups

• A/B tests

• natural experiments

• difference-in-differences

• matched comparisons

The method changes depending on the decision context.

The causal effect is essentially:

{Outcome with intervention} − {estimated outcome without intervention}

This shifts the analytical process from:

“Did the metric move?”

to:

“Did our intervention create incremental change?”

Consider eBay.

eBay was spending heavily on paid search advertising.

Its data showed an obvious relationship:

Paid search clicks → purchases

If you looked at conventional advertising metrics, the conclusion appeared straightforward:

paid search was generating sales.

But there was a fundamental problem.

People who searched for eBay were already more likely to buy from eBay.

So how many of those purchases were actually caused by the advertising?

And how many would have happened anyway?

eBay’s economists tackled this using large-scale field experiments, including turning advertising on and off across different markets to establish a counterfactual. The resulting research was published by Blake, Nosko and Tadelis in Econometrica.

1. What was the intervention?

Paid search advertising.

2. What was the outcome?

Incremental purchases and revenue generated by the advertising.

3. What were the competing explanations?

Some customers were already highly likely to purchase from eBay.

Their search behavior and purchase intent were correlated with the advertising exposure itself.

4. What was the counterfactual?

What would those customers have done if eBay had not shown them the paid search advertisement?

eBay could estimate this by comparing markets where advertising was switched off with comparable markets where it continued.

5. What did the comparison reveal?

The results were striking.

For brand-keyword advertising, the researchers found no measurable short-term benefit.

For non-brand keywords, the effect was more nuanced: new and infrequent users were positively influenced, while frequent users accounted for much of the advertising spend without being materially influenced by the ads. The study concluded that conventional, non-experimental estimates substantially overstated the returns from paid search.

The lesson wasn’t:

“Advertising doesn’t work.”

It was much more sophisticated:

The sales attributed to an intervention are not necessarily the sales caused by it.

That is the difference between attribution and causation.

Strong problem solvers don’t stop at:

“X happened after we did Y.”

They ask:

“Would X have happened anyway?”

The next time an initiative is declared a success, ask:

Compared with what?