Expert Instructor and Course Creator for Embedded Systems, IoT, and Edge AI
In the previous post, we showed how we can substitute our usual ε-greedy policy with a parameterized approximation (often a neural network), we then derived the policy gradient theorem required to optimize this approximator function, and demonstrated how it can be estimated using Monte Carlo sampling. At the very end, we pointed out that the […]
In the previous post, we introduced the breakthrough concept of combining deep learning and reinforcement learning (RL). Instead of recording estimated Q-values in a table, which is intractable for large or continuous state spaces, we approximated those Q-values using a neural network. This simple act spawned the current generation of deep RL, paving the way […]
Previously, we looked at how Q-learning used off-policy temporal difference (TD) updates to converge on an optimal policy. This reinforcement learning (RL) algorithm works surprisingly well, but it requires the environment to have relatively small, discrete state and action spaces. The Q-table can quickly grow to intractable sizes with environments that have a large number […]
One of the biggest breakthroughs in reinforcement learning (RL) occurred in 1989 with Chris Watkins’s paper, Learning from Delayed Rewards. In it, he proposed Q-learning, which decouples the experience gathered from the policy update. In other words, the agent can collect experience using one policy (called the behavior policy) while updating a different policy (called […]
In the previous post, we saw how temporal difference (TD) learning updated value predictions in the middle of an episode rather than waiting to the very end, like we do with Monte Carlo (MC) methods. If you recall, the TD(0) algorithm updates value estimates after every step using a single reward in order to bootstrap […]
Temporal Difference (TD) learning is one of the foundational concepts in reinforcement learning (RL). It combines the notion of updating estimates before the final outcome is known, similar to how dynamic programming (DP) works, with the notion of learning directly from experience, like we saw with the Monte Carlo (MC) methods in part 7. MC […]
In the previous post, we defined a policy, provided the foundational concept of a Markov Decision Process (MDP), and talked about trajectories. We’re going to combine these concepts with the idea of future discounted returns to create value functions. We now introduce two new concepts: state-value function (given by V(s)) that attempts to estimate the […]
In the third part of this reinforcement learning (RL) series, we’re going to give a formal definition for a policy and then conceptualize how actions and states play out in a trajectory. While we discussed rewards and returns in the previous post, we’re going to see how Markov Decision Processes (MDPs) provide the underlying foundation […]
In this second post on reinforcement learning (RL), we build on the introduction from part 1 by revisiting the idea of a reward and building up to the idea of discounted returns. Recall that the goal of RL is to maximize the rewards earned by the agent over time. We’re going to discuss three main […]
Reinforcement learning (RL) is a field of study within machine learning (ML) concerned with developing intelligent agents that take actions in dynamic environments in order to maximize their rewards. RL has gained a lot of popularity in the past few years, most notably in robotics, where big-name companies are using it to create robust locomotion […]