I’ve come to think of varlen primarily as the most efficient FlashAttention variant for training (it’s not used for generating tokens) because it handles our technique of “processing a batch of examples” more efficiently–by treating them as one long concatenated sequence, rather than adding an additional “batch dimension” to the input tensors, which handles varying sequence lengths less naturally.…
Recent models like DeepSeek-V3 and Moonshot’s Kimi-K2, built using Multihead Latent Attention (MLA), have shown that constraining the input spaces of attention heads can be both effective and efficient. They project the input token vector–size 7,168–down to just 512 dimensions for keys and values, and to 1,536 for queries. Despite this aggressive compression, performance holds up well enough to…
Transformers store, retrieve, and modify data along different feature directions in their model space, via projections . I’m finding that building some better intuition around what this actually means can be a powerful tool for reasoning about LLM architecture. Probably the most intriguing quality of ‘feature directions’ is that a model with an embedding size of 4,096 is able to stuff more than 4K…
Multihead Latent Attention (MLA), introduced by DeepSeek in their V2 model, is an alternative to standard attention (and other variants such as MQA and GQA) which dramatically reduces memory bandwidth requirements for the attention calculations. Overview “Reducing bandwidth” means cutting down the number of memory reads required to perform the overall attention calculation. Standard multihead…
What had me most excited about the merged matrix perspective (and perhaps overly so) was that the patterns and messages are in model space , the same dimension as the vocabulary. By applying the appropriate layer normalization to the vocabulary, it’s possible to compare the patterns and messages to the vocabulary! This is limited to “semantic” behaviors (i.e., relating to the meaning of words),…
Something I find really helpful about this merged-matrix perspective is that it puts everything in “model space”. The patterns and messages and their projection matrices all have the same length as the word embeddings. Once you view attention this way, it becomes clear that the entire transformer process is additive. The output word vector is nothing more than the input embedding plus a weighted…
When you reduce Attention down to two matrices instead of four, the pattern and message vectors represent a more familiar architecture–they form a neural network, whose neurons are created dynamically at inference time from the tokens. This draws a nice parallel between the Feed Forward Neural Network and this “Attention Head Neural Network”. The FFN is a large “static” neural network whose input…
One potential benefit for this merged perspective is that it lets us begin our research into Transformer efficiency from a “more fundamental” definition of Attention. We know we have (at least) three issues to resolve: Compute Cost - $W^P_i$ and $W^M_i$ are expensive to project on to, Cache Size - We want to cache the calculated patterns and messages, but they’re very large, and this becomes a…
In the previous post, we looked at how our tendency to think of Attention in terms of large matrix multiplications obscures some key insights, and we rolled back the GPU optimizations in order to reveal them (recapped in the next section). In this post, we’ll continue this process of reformulating Attention in a way that’s mathematically equivalent to the original, but emphasizes conceptual…
In this post, we’ll look at how a tiny bit of algebra suddenly opens up a wealth of insight. I want to clarify up front that none of the “rearranging” that I do in this post is intended to change how we implement Attention. It’s about exposing underlying operations that are already there –we just haven’t noticed them because of the (important!) emphasis we place on finding computationally…
I recently had a series of “aha!” moments around the Attention equations that’s lead to some exciting weeks of research and insight. The core revelation is that the way we’ve been taught to think about Transformers reflects an emphasis on computational efficiency and GPU optimization, and that if we step back from the implementation details we can arrive at a cleaner, more intuitive…
I was curious to dig in to these DeepSeek models that have been making waves and breaking the stock market these past couple weeks (since DeepSeek-R1 was announced in late January ‘25). i. Architecture of a Reasoning Model I dug into the details of R1’s model architecture, thinking it might hold the answers to how reasoning works… I figured out pretty quickly that it doesn’t. (As I covered in my…
I’ve been really curious to know what’s actually happening behind the scenes when you ask OpenAI’s o1 model a question. From what they do show us, it seems pretty clear that the model is breaking the question down, tackling the problem in steps, reviewing its own work, etc. But considering how long the responses took to generate, and that the process was kept a secret, I assumed there must be…
This blog post builds upon a community notebook from Unsloth titled Mistral 7B Text Completion - Raw Text Training Full Example . I went through the original in one of my “Weekly Walkthrough” sessions, learned more about Continued Pre-Training (CPT) in the process, and decided to create a post from it with more code comments and to share the insights we gathered. The code in the notebook remains…
I’ve been curious to see just how well today’s enormous, Decoder-only models–designed for text generation–perform at traditional NLP tasks such as classifying text. These simpler tasks rely purely on the model’s ability to understand the input text, rather than generate a continuation, and Encoder-only architectures like BERT have a strong advantage in this. In this Notebook, I’ve taken my…
An in-depth tutorial on the algorithm and paper, including a pseudo-implementation in Python. by Chris McCormick S1. Introduction QLoRA is really about a technique called “4-bit quantization”. It’s not a different version of “LoRA”, so the title can be confusing. We’ll see why LoRA is relevant, but let’s start with the more important part–representing weights with 4-bit values. Getting LLM weights…
Updated March 2026 This post has become a popular resource for understanding the Colab GPU ecosystem, and especially with the recent addition of the H100 and G4 options, I think it’s long overdue for an update. I originally put this together while researching the topic of fine-tuning Large Language Models (LLMs) on a single GPU in Colab (a challenging feat!), comparing both the free (Tesla T4) and…
▂▂▂▂▂▂▂▂▂▂▂▂ I. Introduction A friend of mine was taking a college course in political science with a ton of assigned reading material, and found that ChatGPT could produce helpful summaries (and in case you’re wondering, the summaries are intended as an additional learning aid, rather than a replacement for doing the reading 😜). There were a few challenges to trying to use ChatGPT for this,…
I’ve studied the samplers a bit and done some of my own experiments with them, and I’ve arrived at some tentative conclusions for what to do with them. Essentially, there are already so many different settings to play with (not to mention the prompt!), and I don’t think exploring the different samplers will net you much. It seems better to simplify this choice, especially as you’re just starting…
The Classifier-Free Guidance Scale, or “CFG Scale”, is a number (typically somewhere between 7.0 to 13.0) that’s described as controlling how much influence your input prompt has over the resulting generation. It’s easy to misinterpret that explanation, though, and to expect the wrong thing from this parameter, so let’s look at CFG scale in more detail. What Does “Guidance” Mean? Stable Diffusion…
In this series of posts I’ll be explaining the most common settings in stable diffusion generation tools, using DreamStudio and Automatic1111 as the examples. This first post will cover the steps slider and the seed value, and then further posts will cover the “cfg scale”, and “sampler”. Steps Let’s start with the step count. In DreamStudio, this is labeled “Steps”, with the description “How many…
The ability for a computer to generate art from nothing but a written description is fascinating! I know that I, for one, would be desperately curious to see what’s actually going on “under the hood” that would make this possible, so I wanted to do what I can here to provide a less superficial explanation of what’s going on even for those who aren’t familiar with the concepts in artificial…
Running Stable Diffusion by providing both a prompt and an initial image (a.k.a.” img2img ” diffusion) can be a powerful technique for creating AI art. In this tutorial I’ll cover: A few ways this technique can be useful in practice What’s actually happening inside the model when you supply an input image. By Chris McCormick Contents Contents 1. How it’s used 1.1. Art from a Doodle 1.2. New Art…
I think AI art inspires awe in us because: It’s incredibly imaginative . The artistic technique is masterful. It was created by an AI . Unfortunately, what it so masterfully generates also tends to be very incoherent . In my experience, you should probably decline any offers for a ride in a stable diffusion fighter jet… 😜 Browsing libraries of generated imagery, the most popular (and successful)…
In this tutorial we’ll look at the topic of classifying text with BERT, but where we also have additional numerical or categorical features that we want to use to improve our predictions. To help motivate our discussion, we’ll be working with a dataset of about 23k clothing reviews. For each review, we have the review text, but also additional information such as: The age of the reviewer…
In this post, we’ll create a very simple question answering system that, given a natural language question, returns the most likely answers from a corpus of documents. This represents an easy to follow and low-effort example of a question answering system that can still be of practical use for you own application if you choose to swap in your own dataset or build something more elaborate on top of…
In this post I wanted to share some of the main themes from NLP over the past year, as well as a few interesting highlights from NeurIPS, the largest annual machine learning conference. by Nick Ryan Contents Contents Trends in NLP NeurIPS NLP EfficientQA Competition Chatbot Tutorial Bias and Ethics New Directions Trends in NLP There was an explosion of new language models in 2020. The Huggingface…
Up to this point, our tutorials have focused almost exclusively on NLP applications using the English language. While the general algorithms and ideas extend to all languages, the huge number of resources that support English language NLP do not extend to all languages. For example, BERT and BERT-like models are an incredibly powerful tool, but model releases are almost always in English, perhaps…
In this blog post / Notebook, I’ll demonstrate how to dramatically increase BERT’s training time by creating batches of samples with different sequence lengths. I learned this technique from Michaël Benesty in his excellent blog post here , and used key pieces of his implementation ( here ) in this Notebook. Michaël’s code is designed to make use of the new Trainer class in the transformers…
While working on my recent Multi-Class Classification Example , I was having trouble with running out of memory on the GPU in Colab–a pretty frustrating issue! There were actually three parameters at play which could lead to running out memory: My choice of training batch size ( batch_size ) My choice of sequence length ( max_len ) Which Tesla GPU Colab gave me! This forced me to pay more…
If your text data is domain specific (e.g. legal, financial, academic, industry-specific) or otherwise different from the “standard” text corpus used to train BERT and other langauge models you might want to consider either continuing to train BERT with some of your text data or looking for a domain-specific language model. Faced with the issue mentioned above, a number of researchers have created…
In conjunction with our tutorial for fine-tuning BERT on Named Entity Recognition (NER) tasks here , we wanted to provide some practical guidance and resources for building your own NER application since fine-tuning BERT may not be the best solution for every NER application. In this post, we will: Discuss when it might be appropriate to use an off-the-shelf library vs. training / fine-tuning your…
by Chris McCormick As I’ve been doing all of this research into BERT, I’ve been really curious–just how much trivia does BERT know? We use BERT for it’s impressive knowledge of language, but how many factoids are encoded in there along with all of the language understanding? It turns out, kind of a lot! We’re going to look at some fun examples in this post. Now, BERT can’t generate text, so we…
What does it mean for BERT to achieve “human-level performance on Question Answering”? Is BERT the greatest search engine ever, able to find the answer to any question we pose it? In Part 1 of this post / notebook, I’ll explain what it really means to apply BERT to QA, and illustrate the details. Part 2 contains example code–we’ll be downloading a model that’s already been fine-tuned for question…
1. Introduction In this “research notes” blog post, and the ones that follow it, I’ll be sharing what I am learning about BERT, as well as identifying the areas where I am perhaps confused or where I still need to learn more. I think that the NLP community is currently missing an in-depth tutorial on the BERT model which does not require extensive background knowledge in LSTMs and Attention. More…
By Chris McCormick and Nick Ryan In this post we take a look at an important NLP benchmark used to evaluate BERT and other transfer learning models! Introduction The General Language Understanding Evaluation benchmark (GLUE) is a collection of datasets used for training, evaluating, and analyzing NLP models relative to one another, with the goal of driving “research in the development of general…
If your first foray into Machine Learning was with Andrew Ng’s popular Coursera course (which is where I started back in 2012!), then you learned the fundamentals of Machine Learning using example code in “Octave” (the open-source version of Matlab). Octave is great for expressing linear algebra operations cleanly, and (as I hear it) for being easier for non-programmers to get going with. It’s…
By Chris McCormick and Nick Ryan In this tutorial, I’ll show you how to finetune the pretrained XLNet model with the huggingface PyTorch library to quickly produce a classifier for text classification. Introduction (This post follows the previous post on finetuning BERT very closely, but uses the updated interface of the huggingface library (pytorch-transformers) and customizes the input for use…
By Chris McCormick and Nick Ryan Revised on 3/20/20 - Switched to tokenizer.encode_plus and added validation loss. See Revision History at the end for details. In this tutorial I’ll show you how to use BERT with the huggingface PyTorch library to quickly and efficiently fine-tune a model to get near state of the art performance in sentence classification. More broadly, I describe the practical…
In this post, I take an in-depth look at word embeddings produced by Google’s BERT and show you how to get started with BERT by producing your own word embeddings. This post is presented in two forms–as a blog post here and as a Colab notebook here . The content is identical in both, but: The blog post format may be easier to read, and includes a comments section for discussion. The Colab Notebook…
I’m proud to announce that I’ve published my first eBook, The Inner Workings of word2vec . It includes all of the material in the popular word2vec tutorial on my blog, and goes deeper with additional topics like CBOW and Hierarchical Softmax. I’ve also created example code to go along with the book that exposes the algorithm details and let’s you see them in action. If you have any questions,…
In this article, I wanted to share about a trend that’s occurred over the past few years of using the word2vec model on not just natural language tasks, but on recommender systems as well. The key principle behind word2vec is the notion that the meaning of a word can be inferred from it’s context–what words tend to be around it. To abstract that a bit, text is really just a sequence of words, and…
In part 1 of this tutorial , I described the most basic form of a product quantizer. In this post, I’ll be explaining the IndexIVFPQ index from the FAISS library , which uses a product quantizer as well as a couple additional techniques introduced in their 2011 paper . Here is a brief summary of the two added features, followed by more detailed explanations. Inverted File Index (IVF) The IVF is…
Exhaustive Search with Approximate Distances Explanation by Example Dataset Compression Nearest Neighbor Search Compression Terminology Pre-filtering A product quantizer is a type of “vector quantizer” (I’ll explain what that means later on!) which can be used to accelerate approximate nearest neighbor search. They’re of particular interest because they are a key element of the popular Facebook AI…
This post was written in my role as a researcher at Nearist, and will soon be on the Nearist website as well. This article is the first in a series comparing different available methods for accelerating large-scale k-Nearest Neighbor searches on high-dimensional vectors (i.e., 100 components or more). The emphasis here is on practicality versus novelty–that is, we’re focusing on solutions which…
I recently created a project on GitHub called wiki-sim-search where I used gensim to perform concept searches on English Wikipedia. gensim includes a script, make_wikicorpus.py , which converts all of Wikipedia into vectors. They’ve also got a nice tutorial on using it here . I started from this gensim script and modified it heavily to comment and organize it, and achieve some more insight into…
I’ve recently needed to perform a benchmarking experiment with k-NN in C++, so I found mlpack as what appears to be a popular and high-performance machine learning library in C++. I’m not a very strong Linux user (though I’m working on it!), so I actually had a lot of trouble getting up and going with mlpack, despite their documentation. In this guide, I’ll cover the steps needed to get up and…
In part 2 of the word2vec tutorial (here’s part 1 ), I’ll cover a few additional modifications to the basic skip-gram model which are important for actually making it feasible to train. When you read the tutorial on the skip-gram model for Word2Vec, you may have noticed something–it’s a huge neural network! In the example I gave, we had word vectors with 300 components, and a vocabulary of 10,000…
DBSCAN is a popular clustering algorithm which is fundamentally very different from k-means. In k-means clustering, each cluster is represented by a centroid, and points are assigned to whichever centroid they are closest to. In DBSCAN, there are no centroids, and clusters are formed by linking nearby points to one another. k-means requires specifying the number of clusters, ‘k’. DBSCAN does not,…
In this post I’m sharing a technique I’ve found for showing which words in a piece of text contribute most to its similarity with another piece of text when using Latent Semantic Indexing (LSI) to represent the two documents. This has proven valuable to me in debugging bad search results from “concept search” using LSI. You’ll find the equations for the technique as well as example Python code. My…