Information Theory Cheat-Sheet

Entropy, divergence, mutual information, etc. are central concepts in statistical machine learning. This post ties them together in a short collection of elementary information theoretic results. Below, we consider random variables $\mathrm{X}, \mathrm{Y}$ that take values in some discrete sets $\mathcal{X}$ and $\mathcal{Y}$. We denote, respectively, $p_{\tiny\mathrm{X}}\in\Delta_\mathcal{X}$ and…