Semi-Gradient Sarsa and the Average Reward Setting in RL (S&B Ch. 10)

Let’s dive into the control problem now with parametric approximation of the action-value function $\hat{q}(s, a, \mathbf{w}) \approx q_{*}(s, a)$, where $\mathbf{w} \in \mathbb{R}^d$ is a finite-dimensional weight vector. We’ll focus on semi-gradient Sarsa , the natural extension of semi-gradient TD(0) to action values and to on-policy control. We’ll look at this extension in both the episodic…