Exploring Deep Learning.
Mixture of Experts (MoE) is one of the main techniques used to scale modern language models without making every token pay the full computational cost of the model. The basic idea is surprisingly simple: instead of sending every token through one enormous feed-forward network, we split it into many smaller expert networks and only activate a few experts for each token. These notes walk through the…
Attention mechanisms have evolved considerably to make transformer inference faster and more memory-efficient. These notes trace that evolution: from vanilla self-attention , through KV caching , to memory-saving variants like MQA , GQA , and MLA , and finally to sparse attention methods like SWA and Deepseek Sparse Attention (DSA) . Table of Contents Self-Attention Masked (Causal) Self-Attention…
Almost all the algorithms/methods covered so far have been action-value methods (except gradient-bandit algorithms, Section 2.8 ). Action-value methods learn the values of actions and then derive the policy thereafter to select actions based on their estimated action values. Here, we explicitly learn a parametrized policy that can select actions without consulting a value function. A value…
Transformers are a sequence-to-sequence model : given an input sequence, produce an output sequence. Architecture: an Encoder processes the input; a Decoder generates the output autoregressively. \[\text{(En) "I am sorry"} \xrightarrow{\text{Encoder}} \xrightarrow{\text{Decoder}} \texttt{<start>}\ \text{Je suis désolé}\ \texttt{<end>}\] Autoregressive : the decoder generates one token at a time,…
Meta’s unified model for promptable image and video segmentation. A foundation model for solving promptable visual segmentation in images & videos . Built a data engine to collect the largest video segmentation dataset to date. Model : Simple transformer architecture with streaming memory for real-time video processing. Trained on a wide range of tasks: video segmentation and image segmentation.…
Muon stands for M oment U m O rthogonalized by N ewton-Schulz and was invented by Keller Jordan . The key idea: Instead of applying Adam-style per-element adaptive updates to model parameters, Muon orthogonalizes the momentum matrix before using it as the update direction. Table of Contents Adam Optimizer Matrix Orthogonalization Newton-Schulz 5 Iteration Muon QK-Clip Multihead Latent Attention…
Earlier this year, I decided to force myself to read more. Not a New Year’s resolution, because those never last. The reason is that growing up as a child and young teenager, reading often felt like punishment. My mum required my siblings and me to read a certain number of pages from a designated book every day throughout elementary school. Missing a day meant mandatory punishment. In boarding…
Eligibility traces are one of the basic mechanisms of RL that unify and generalize TD and Monte Carlo (MC) methods. TD methods augmented with eligibility traces produce a family of methods spanning a range from MC methods at one end ($\lambda = 1$) to one-step TD (TD(0)) methods at the other end ($\lambda = 0$). With eligibility traces, MC methods can be implemented online and on continuing…
Let’s discuss the extension of off-policy methods from the tabular case (Ch. 6 & 7) to function approximation. We’ll explore the convergence problems, the theory of linear function approximation, the notion of learnability, and stronger convergence off-policy algorithms. Off-policy learning with function approximation has 2 challenges: Finding the target of the update. The off-policy distribution…
Let’s dive into the control problem now with parametric approximation of the action-value function $\hat{q}(s, a, \mathbf{w}) \approx q_{*}(s, a)$, where $\mathbf{w} \in \mathbb{R}^d$ is a finite-dimensional weight vector. We’ll focus on semi-gradient Sarsa , the natural extension of semi-gradient TD(0) to action values and to on-policy control. We’ll look at this extension in both the episodic…