How Many Random Neurons Buy a Trained One?

Freeze a bank of Yat units at random initialization, noise prototypes that will never move, and train nothing but the linear readout on top. This should not work, and the reason it does is one of the oldest facts in the kernel literature wearing a new activation: a frozen random layer is a Monte Carlo estimate of a kernel, one neuron per sample, converging at the square-root rate. This post…