The Deadly Triad in RL: Off-Policy Learning with Function Approximation (S&B Ch. 11)

Let’s discuss the extension of off-policy methods from the tabular case (Ch. 6 & 7) to function approximation. We’ll explore the convergence problems, the theory of linear function approximation, the notion of learnability, and stronger convergence off-policy algorithms. Off-policy learning with function approximation has 2 challenges: Finding the target of the update. The off-policy distribution…