Recent content on CruxPonent
For its own sake or as part of a policy iteration scheme, evaluating policies is an important matter. This post is concerned with off-policy policy evaluation, or evaluating policies with data generated by others. The ambition is to provide a smooth progression towards the Retrace estimator and some of its extensions, with a focus on each operator’s properties and stochastic approximation…
Coming from a control background, applying RL to text generation is hardly intuitive. Is there an actual, non-degenerate dynamical system at play here? What is the concrete novelty behind the shiny LLM post-training algorithms? This post is an attempt to answer those questions by providing a semiformal derivation of the celebrated GRPO algorithm through a contextual bandit lens. $\quad$ The focus…
This post explores the theory of regularised MDPs beyond entropic regularisation (which we covered in an older post). We will introduce convex regularisation of the classical Bellman operators and study the induced regularised policy iteration algorithms. On the way, we will tie some links with several popular algorithms. This post is mostly a good excuse to refresh some convex optimisation…
Entropy, divergence, mutual information, etc. are central concepts in statistical machine learning. This post ties them together in a short collection of elementary information theoretic results. Below, we consider random variables $\mathrm{X}, \mathrm{Y}$ that take values in some discrete sets $\mathcal{X}$ and $\mathcal{Y}$. We denote, respectively, $p_{\tiny\mathrm{X}}\in\Delta_\mathcal{X}$ and…
For better or for worse, proximal policy optimisation (PPO) algorithms and its variants are dominating the RL landscape these days. This post aims at retracing their journey, from foundational concepts to LLM-savy innovations. We will start this saga on the theoretical trail, which we will progressively abandon to pay closer attention to algorithmic aspects. $\quad$ We rely on standard notations…
The goal of this post is to explore from first principles the learning of belief models in partially observable MDPs. We will start with a quick refresher on variational inference, and apply it to state estimation in POMDPs. Specifically, we will derive the update rule used to train Dreamer-like models. Variational Inference In this post, we are interested in latent variable models. We consider…
In this second post of this series, we take a break from successor measures to focus on successor features . We will first review the use of a generalised policy improvement mechanism that can efficiently leverage the successor features of existing policies to enable zero-shot transfer to new tasks. We will then discuss the generalisation to universal successor features approximations, allowing…
Thanks to its relative simplicity and conciseness, the discounted approach to control in MDPs has come to largely prevail in the RL theory and practice landscape. Departing from the myopic nature of discounted control, we study here the average-reward objective which focuses on long-term, steady state rewards. To start gently, we will limit ourselves to establishing Bellman equations for policy…
The main promise of unsupervised RL is test-time adaptation to newly specified reward functions. This requires a systemic untangling of reward and dynamics in traditional RL tools. In this first post of a short series, we see how this can be done via the concept of successor states and successor representations. We will focus on policy evaluation, leaving control for a follow-up. One over-arching…
This post is interested in state estimation in HMMs: filtering, prediction and smoothing. We will introduce state estimation as the solution of an optimisation problem, and prove the celebrated recursive updates for each inference use-case. A special attention will be given to HMM filters (and how they easily generalise to the celebrated Kalman filters). $\quad$ The reader interested about…