Dropout as Shared-Parameter Bagging

I worked through Chapter 7 of Deep Learning last week, and the bagging interpretation of dropout was the most useful part. The shortcut explanation treats dropout as random noise that helps regularization. Chapter 7 gives a more precise model: dropout works like a relatively inexpensive form of model averaging, close in spirit to bagging.

Dropout approximates model averaging

Bagging is powerful because it reduces variance by averaging predictions from many models trained on perturbed data. The trade-off is straightforward: training and serving many separate models is expensive. Dropout gets part of the same effect by sampling a different thinned network on each minibatch update while all of those subnetworks share parameters. During inference, scaling activations gives an efficient approximation to averaging over that family.

That makes dropout easier to explain. It is not just random noise for regularization. It is a cheap way to get some of the benefits of averaging over many related models without training each one separately.

Changing masks discourages co-adaptation

The bagging interpretation also makes another part of dropout easier to think about: the apparent paradox that it can widen the set of subnetworks seen during training while still regularizing. Each binary mask defines a different thinned network, but those networks share weights. At the same time, any one unit cannot rely on a specific partner always being present, so representations are pushed to be useful across many subnet configurations.

That pressure reduces fragile co-adaptation. Features that only work in one narrow pathway get penalized indirectly, while more robust features survive across many sampled masks. I like this framing because it explains dropout through the training setup itself, not just through the shortcut phrase "add noise."

The cleaner frame is dropout as shared-parameter ensemble training, not just noise. That explains why it can regularize effectively without feeling mysterious. More broadly, it is a useful example of a deep learning technique that is simple at the rule level but grounded in a more specific training setup than the shortcut explanation suggests.