A little experiment in evading AI detection
Notes from trying to make an AI-drafted series sound human.
Nick Gustafson
Writing by Nick Gustafson — software engineer and data scientist. Notes on machine learning, the systems behind it, and the things worth understanding deeply.
Latest posts
My Throw Decides My Aim
A D-A-D song leads us to latent space thinking.
Vectors, Matrices, and the Spaces They Live In
Vectors as lists of activations, matrix multiplication as a linear map, and why every neural network operation bottoms out in matmuls.
Norms, Dot Products, and Similarity
How cosine similarity, L2 distance, and projections work, and why they show up everywhere from attention scores to embedding retrieval.
Distributions, Softmax, and the Chain Rule of Words
Softmax, categorical distributions, Bayes' rule, and the chain rule of probability — the four tools that make language modeling a well-defined math problem.
Cross-Entropy, KL Divergence, and What Loss Functions Measure
Why cross-entropy is the standard LM loss, what it actually measures about two distributions, and how it connects to perplexity.
Gradients and How Machines Learn
What a gradient is, why it points uphill, how backpropagation computes one efficiently via the chain rule, and what SGD does with it.
Optimizers: Momentum, Adam, and Learning Rate Schedules
Why vanilla SGD is too slow, how Adam adapts per-parameter, and how warmup and cosine decay shape training dynamics.
GPUs, Floating Point, and Why Precision Matters
IEEE 754, the difference between fp32/fp16/bfloat16, why mixed-precision training works, and the basics of GPU parallelism.
A Short Prehistory of Statistical NLP
The arc from rule-based systems through statistical MT and log-linear models to neural approaches, giving you historical context for everything that follows.
Language Modeling as Next-Token Prediction
Why 'predict the next word' is a surprisingly powerful training objective, and how it connects to the chain rule of probability.
N-gram Models and the Curse of Sparsity
Counting-based language models, Markov assumptions, smoothing techniques, and why sparsity made n-grams hit a hard ceiling.
Word2vec and the Embedding Revolution
How skip-gram and CBOW learn dense word vectors, why 'king - man + woman = queen' works, and what distributional semantics actually means.
GloVe, FastText, and the Embedding Zoo
How GloVe combines count-based and predictive methods, how FastText handles subwords, and the trade-offs across embedding approaches.
Recurrent Neural Networks and Sequence Modeling
The RNN equations, how hidden states carry information forward, why vanilla RNNs suffer from vanishing gradients, and what that means in practice.
LSTMs, GRUs, and Gated Memory
How gates solve the vanishing gradient problem, the difference between LSTM and GRU cells, and why gated architectures dominated NLP for five years.
Seq2seq, Bahdanau Attention, and Why Recurrence Hit a Wall
Encoder-decoder architectures, how additive attention let models align source and target tokens, and why sequential processing fundamentally bottlenecked scale.
Unicode, Bytes, and What Text Actually Is
UTF-8 encoding, code points vs. grapheme clusters, and why 'one character' is a surprisingly ambiguous concept for a model.
Byte-Pair Encoding from Scratch
The BPE merge algorithm step by step, implemented from scratch, showing how a raw byte stream becomes a vocabulary.
WordPiece, Unigram, and SentencePiece
How WordPiece (BERT) and Unigram (T5) differ from BPE, why SentencePiece operates on raw text, and when each approach wins.
Vocabulary Size, Merge Order, and Fertility
How vocabulary size affects model capacity and sequence length, what token fertility measures, and the practical trade-offs behind 32k vs. 128k vs. 200k vocabularies.
The Embedding Table and Its Geometry
How token IDs become vectors via a lookup table, why the embedding matrix is a learned linear map, and how embedding and unembedding layers relate.
Special Tokens, Chat Templates, and Input Formatting
BOS/EOS/PAD tokens, how chat-template formatting encodes multi-turn structure, system prompts, and tool-call schemas into a flat token stream.
Packing, Masking, and Tokenization as a Model Interface
Sequence packing for training efficiency, how attention masks prevent cross-contamination, and why tokenizer choices constrain everything downstream.
Self-Attention: Q, K, V from First Principles
Queries, keys, and values as learned linear projections, how scaled dot-product computes soft token-to-token lookups, and exactly why we divide by sqrt(d_k).
Multi-Head Attention and Representation Subspaces
Why multiple attention heads let the model attend to different relationship types in parallel, how heads partition the embedding dimension, and what the output projection does.
Causal Masking and the Autoregressive Constraint
Why decoder models must not attend to future tokens, how the triangular mask enforces this, and how masking shapes both training and generation.
Positional Encodings: Sinusoidal, Learned, RoPE, and ALiBi
Why attention is position-agnostic by default, how each encoding scheme injects order, why RoPE became dominant, and how ALiBi achieves length extrapolation.
The Feed-Forward Block as Key-Value Memory
The two-layer MLP that follows attention, why it can be interpreted as a learned key-value store over concepts, and what the intermediate dimension controls.
Layer Normalization: Pre-Norm, Post-Norm, and RMSNorm
Why normalization is essential for deep residual networks, how pre-norm stabilizes training, and why RMSNorm replaced LayerNorm in modern architectures.
Decoder-Only vs. Encoder-Decoder: Architecture Trade-offs
The structural differences between GPT-style and T5-style models, why decoder-only won for generative LLMs, and where encoder-decoder still shines.
A Close Reading of ‘Attention Is All You Need’
The original 2017 paper section by section, connecting each design choice to what you already know, and noting which ideas survived and which were replaced.
Mixture-of-Experts Layers
How MoE replaces the dense FFN with sparse expert routing, the load-balancing problem, and how expert-choice routing solves it.
Training View vs. Inference View
Why the model you trained on full sequences behaves completely differently when it generates one token at a time, and what that means for everything that follows.
Prefill vs. Decode: The Two Phases of Inference
How the prompt is processed in parallel (prefill) but generation is strictly sequential (decode), and why this split dominates every performance conversation.
Why One New Token Means One New Row
Walking through the exact matrix operations at each decode step to see why generation is memory-bound, not compute-bound.
The KV Cache from First Principles
Deriving the key-value cache by noticing which attention computations are redundant, then watching memory grow linearly with sequence length.
Sampling Strategies: Temperature, Top-k, Top-p, and Min-p
How each strategy reshapes the probability distribution over the vocabulary, when to combine them, and what 'good randomness' actually looks like.
Speculative Decoding
Using a small draft model to guess multiple tokens at once and a large model to verify them in parallel, turning sequential generation into a bet on acceptance rates.
Continuous Batching
Why naive batching wastes GPU cycles on padding, and how iteration-level scheduling lets new requests join a running batch the moment a slot opens.
Prefix Caching and Prompt Reuse
Sharing KV cache across requests that start with the same system prompt, and the cache-eviction policies that make it practical.
Structured and Constrained Generation
How grammar-guided decoding and JSON-mode work by masking logits at each step, and the tradeoff between constraint strength and generation speed.
What an Inference Engine Actually Does
The concrete responsibilities — graph optimization, memory allocation, scheduling, kernel dispatch — that sit between your model weights and an HTTP response.
Kernel Fusion and the Memory Wall
Why fusing multiple operations into a single GPU kernel reduces memory round-trips, and how FlashAttention is the poster child for this idea.
PagedAttention: Virtual Memory for the KV Cache
Treating KV cache like virtual memory pages so you stop wasting 60-80% of GPU RAM on fragmentation.
Memory Management: Fitting a Model and Deciding Concurrency
A unified picture of where bytes go — weights, activations, KV cache, CUDA context — and how to reason about what fits on your GPU.
Quantization: INT8, INT4, GPTQ, AWQ, and GGUF
Shrinking weight precision to serve larger models on smaller hardware, with an honest look at what quality you lose and when.
Tensor, Pipeline, and Expert Parallelism
Three ways to split a model across multiple GPUs, each with different latency and throughput profiles.
Throughput vs. Latency: Picking Your Tradeoff
Why optimizing for tokens-per-second and optimizing for time-to-first-token are fundamentally in tension, and how batching policy mediates.
Comparing Engines: vLLM, TGI, TensorRT-LLM, llama.cpp, SGLang
A framework for choosing an inference stack based on your actual constraints — hardware, model size, latency budget, and deployment environment.