A blog about things and stuff.
This post provides an overview, implementation, and time complexity analysis of DeepMind's paper Accelerating Large Language Model Decoding with Speculative Sampling . Code for this blog post can be found at github.com/jaymody/speculative-samlping . EDIT (Apr 13th, 2023): Updated code and time complexity to avoid the extra forward pass of the draft model (credits to KexinFeng ). Autoregressive…
In this post, we'll implement a GPT from scratch in just 60 lines of numpy . We'll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text. Note: This post assumes familiarity with Python, NumPy, and some basic experience with neural networks. This implementation is for educational purposes, so it's missing lots of features/improvements on…
In this post, we'll take a look at softmax and cross entropy loss, two very common mathematical functions used in deep learning. We'll see that naive implementations are numerically unstable, and then we'll derive implementations that are numerically stable. Symbols (x ): Input vector of dimensionality (d ). (y ): Correct class, an integer on the range (y in [1 ldots K] ). ( hat{y} ): Raw outputs…
ChatGPT and other large language models use a special type of neural network called the transformer. The transformer defining feature is the attention mechanism. Attention is defined by the equation: [ text{attention}(Q, K, V) = text{softmax}( frac{QK^T}{ sqrt{d_k}})V ] Attention can come in different forms, but this version of attention (known as scaled dot product attention) was first proposed…
Background A distance matrix is a square matrix that captures the pairwise distances between a set of vectors. More formally: Given a set of vectors (v_1, v_2, ... v_n ) and it's distance matrix ( text{dist} ), the element ( text{dist}_{ij} ) in the matrix would represent the distance between (v_i ) and (v_j ). Notice, this means the matrix is symmetric since ( text{dist}_{ij} = text{dist}_{ji} ),…