Some Notes on Gradient Estimation

Assume we have a scalar function $f(x)$ of interest, such as a reward we want to maximise or a loss we want to minimise, and that $x$ is drawn from a distribution $p_{\theta}(x)$ parameterised by $\theta$. A natural quantity to study is the expected value of $f$ under this distribution, \[J(\theta) = \mathbb{E}_{x \sim p_{\theta}(x)}[f(x)],\] and we would like to optimise $J(\theta)$ with respect…