Eligibility Traces Explained: TD(λ), Sarsa(λ), and the λ-Return (S&B Ch. 12)

Eligibility traces are one of the basic mechanisms of RL that unify and generalize TD and Monte Carlo (MC) methods. TD methods augmented with eligibility traces produce a family of methods spanning a range from MC methods at one end ($\lambda = 1$) to one-step TD (TD(0)) methods at the other end ($\lambda = 0$). With eligibility traces, MC methods can be implemented online and on continuing…