The Shrinkage Hidden Inside Model Averaging

You have built a predictive model. You ran variable selection — maybe lasso, maybe spike-and-slab, maybe just stepwise regression with some ad hoc stopping rule — and arrived at a specification that includes five predictors out of twenty candidates. The backtest looks reasonable. You deploy it.

A month later, you wonder: what if you picked the wrong five? You know the posterior probability of your chosen model was perhaps 35%. That means there is a 65% chance that some other combination of predictors would have been correct. And yet your confidence intervals, your risk estimates, your position sizes — all of them condition on that one model as if it were certain. You have hidden model uncertainty under the rug, and your reported precision is a fiction.

This is the problem that Bayesian model averaging addresses. Instead of committing to one model and pretending you are certain, you weigh all plausible models by their posterior probability and let the predictions reflect the spread. The intuition is straightforward. The mechanics, less so. But there is a way to understand what BMA actually does to your coefficient estimates that makes the whole enterprise transparent: BMA is shrinkage.

Practitioners tend to think of shrinkage and model selection as two distinct tools. Shrinkage pulls all coefficients toward zero, leaving you with a dense model where every predictor contributes something. Model selection zeros out coefficients entirely, leaving you with a sparse model where a few predictors do all the work. Ridge and lasso, one might say, sit at opposite ends of a spectrum. The former shrinks uniformly; the latter selects.

This framing is clean, but it obscures a deeper connection. When you run Bayesian model averaging over a set of sparse models, the averaged estimate is dense. Each individual model includes some predictors and excludes others. But because you are weighting across models, every predictor receives some contribution from the models that include it. The more models that include a predictor, the larger its weight in the average.

Here is where the interpretation of shrinkage becomes crisp. The BMA posterior mean for any coefficient can be written as the product of two terms: the posterior inclusion probability (PIP) and the conditional mean of the coefficient given that it is included. The PIP tells you how much evidence the data provide that the predictor belongs in the model. The conditional mean tells you what the coefficient would be if you knew the predictor were relevant. Multiply them together, and you have the BMA estimate.

Suppose three models survive your model comparison:

The BMA estimate is the weighted average across all models:

Now decompose this into the two terms. The posterior inclusion probability is the total weight on models that include the predictor:

The conditional mean given inclusion is the weighted average among models that include j, with weights renormalised to sum to one:

Multiply them:

Same answer.

The shrinkage is now visible. The conditional mean given inclusion is 0.54 — what you would estimate if you knew the predictor belonged in the model. But you do not know that. The PIP of 0.70 discounts this estimate by the probability of inclusion, pulling it down to 0.38. The 30% posterior weight on Model C, which excludes the predictor entirely, is what does the pulling.

This decomposition makes the shrinkage structure explicit. If a predictor has a PIP near one — meaning almost every visited model includes it — the BMA estimate is essentially the conditional mean. No shrinkage. If the PIP is near zero, the BMA estimate is pulled almost entirely to zero. Full shrinkage. And if the PIP is somewhere in between — 0.70 in the example above — then the estimate is 70% of what it would have been under certainty of inclusion.

This is adaptive shrinkage arising from discrete model uncertainty. Continuous shrinkage priors — the horseshoe, the lasso, and their many variants — achieve something similar through mixing distributions over variances. BMA achieves it by averaging over models, each of which makes a binary inclusion decision. The end result is the same: coefficients with weak evidence get pulled toward zero; coefficients with strong evidence are left alone. But the mechanism differs. Continuous priors use heavy-tailed or sparsity-inducing distributions that resist shrinkage for large signals. BMA uses the posterior model probabilities to determine how much weight to give each model’s estimate.

The comparison lines up cleanly. Ridge-type priors apply the same shrinkage factor to every coefficient — a function of the prior variance relative to the data precision. Global-local shrinkage priors apply a coefficient-specific factor determined by a more adaptive structure. Spike-and-slab makes a hard zero-or-one decision. BMA sits in an interesting middle ground: it arises from discrete inclusion decisions, but the averaging produces a continuous shrinkage factor between zero and one.

There is a practical payoff to understanding BMA as shrinkage. When you average over sparse models, your prediction is dense. Every predictor contributes, weighted by its PIP. This helps explain an empirical regularity that has puzzled macro-forecasters and financial economists: dense shrinkage priors often outperform sparse ones, even when the true data-generating process is itself sparse. One reason is precisely this averaging effect. A sparse model commits fully to its included predictors; if one of them is wrong, the forecast suffers. A dense model hedges its bets, and the diversification across predictors smooths out individual errors. Skepticism pays off when noise dominates.

Model averaging also reveals the variance hidden by model selection. When you condition on a single model, your reported uncertainty reflects only parameter estimation error within that model. BMA adds a second component: between-model variance, capturing how much the coefficient estimates differ across plausible specifications. In applications with genuine model ambiguity, this component can be substantial — 5%, 10%, sometimes more of total posterior variance. Ignoring it leads to credible intervals that are too narrow and decisions that are overconfident.

BMA and shrinkage are not competitors. They are two expressions of the same statistical impulse: regularise the estimate to account for uncertainty, whether that uncertainty lives in the coefficient magnitude or in the model specification. Once you see the PIP as a shrinkage factor, the connection becomes natural. Every posterior inclusion probability is telling you how much to trust a predictor — and how much to pull it back toward zero.

The forecaster who uses ridge regression and the forecaster who runs model averaging are both shrinking. They differ in how they calibrate that shrinkage. Ridge uses a global parameter; model averaging uses the data’s own verdict on which predictors matter. The latter is more adaptive, and in many applications, more accurate. But both deserve to be understood as points on the same continuum.

A simple Python code demonstrating the shrinkage decomposition is available on my GitHub.

For the foundations of BMA, see Hoeting, Madigan, Raftery & Volinsky (1999), "Bayesian Model Averaging: A Tutorial," Statistical Science; and Raftery, Madigan & Hoeting (1997), "Bayesian Model Averaging for Linear Regression Models," Journal of the American Statistical Association.

Tags: Finance, Machine Learning, Bayesian Methods, Forecasting, Econometrics