Paper Review: How much do language models memorize?

Intro Setup Memory definitions Toy Example Prior, dataset, and model Preliminary entropy calculations Total entropy $H(X)$ Conditional entropy $H(X\vert\Theta)$ Memory calculations Mutual Information (Total Memorization) Unintended Memorization (Rote) Intended Memorization (Generalization) Discussion $\text{mem}_I > \text{mem}_U$: what it doesn’t mean Last look at the memory math The rest of the…