How Prompt Caching, RAG, and GGUF actually work

This week I wrote six blogs. Each one takes a single idea and explains it slowly, in very simple words, until a complete beginner understands every single word.

Together they cover the pieces behind modern AI search, RAG, and local inference.

Let’s get started.

Prompt Caching is a technique where the model saves the work it already did for a repeated part of a prompt, so that next time it can reuse that saved work instead of doing it all over again.

In simple words: do the hard work once, then reuse it.

The whole trick is the exact-prefix rule. Put the stable part first, and put the changing part last. If even one character changes early in the prompt, the cache breaks for everything after that change.

A cache write costs a little more, around 1.25 times the normal price. But a cache read is very cheap, around one-tenth of the normal price. That is roughly a 90 percent saving on the repeated part.

Read it here: https://outcomeschool.com/blog/how-does-prompt-caching-work

Token streaming is a technique where the server sends the model’s reply to us piece by piece, as each piece is produced, instead of waiting for the whole reply to be ready.

We have all seen the words appear on the screen one small chunk at a time, like someone is typing the answer live. That live typing effect is token streaming in action.

The technique that carries the tokens is SSE, which stands for Server-Sent Events. The HTTP connection is opened once and kept open, the server sets the Content-Type to text/event-stream, it pushes each token as a data: line, and a final [DONE] marker tells the browser the reply is complete.

This is what makes the time to first token tiny, so the whole app feels fast and responsive.

Read it here: https://outcomeschool.com/blog/how-does-token-streaming-work

A normal database is good at finding exact matches. A vector database is good at finding things that are close in meaning.

The key idea: similar meaning gives similar numbers. Each embedding is a vector, and that is exactly what we store inside a vector database.

But brute force, comparing the query with every single stored vector, is too slow at scale. So here comes Approximate Nearest Neighbour (ANN) to the rescue. It trades a tiny bit of accuracy for a huge gain in speed.

Inside, I break down the three similarity metrics (cosine similarity, dot product, euclidean distance) and the indexing methods that make it fast (HNSW, IVF, and PQ), all with a formula, an example, and a diagram.

Read it here: https://outcomeschool.com/blog/how-does-a-vector-database-work

A Reranker is a model that takes a list of documents and reorders them, putting the most relevant ones at the top for a given question.

In simple words, a Reranker is a smart sorter. It does not search the whole world. It only re-arranges a shortlist that someone else already gathered.

This is the two-stage retrieval idea. The first stage trades precision for speed. The second stage trades speed for precision. Together, they give us the best of both.

The secret is the difference between a bi-encoder and a cross-encoder. The fast bi-encoder shortlists many candidates, and the accurate cross-encoder reorders only that shortlist. The Reranker is the cross-encoder. I also cover ColBERT, the clever middle ground between the two.

Read it here: https://outcomeschool.com/blog/how-does-a-reranker-work

Knowledge Distillation is a technique where we train a small model to copy the behavior of a large model.

The big model is the teacher. The small model is the student. Our goal is to pass the teacher’s knowledge into the student.

The key idea is that the teacher does not just give the right answer. It gives soft labels, a full set of probabilities over all classes. These soft labels carry dark knowledge, which means they reveal how the classes relate to each other, like a cat being a little bit “dog” and nothing like a “car”.

We raise the temperature in the softmax to make this dark knowledge clear, and we train the student with a combined loss until it behaves like the teacher. This is how we shrink the brain into a smaller body while keeping most of the intelligence.

Read it here: https://outcomeschool.com/blog/how-does-knowledge-distillation-work

GGUF is a single file format that stores everything needed to run a large language model for local inference, all in one self-contained file.

The old way kept the weights, the tokenizer, and the settings in separate files. It was fragile, not portable, and slow to load. GGUF is like getting one neatly packed box that has everything inside, clearly labeled, ready to use the moment we open it.

It uses quantization to shrink the weights, storing each weight using fewer bits so big models fit on a normal laptop. Drop from 16 bits to about 4 bits, and a 7 billion weight model shrinks from around 14 gigabytes to roughly 4 gigabytes. I also decode those confusing names like Q4_K_M, and explain how memory mapping makes it load fast.

This is the format behind llama.cpp, Ollama, and LM Studio.

Read it here: https://outcomeschool.com/blog/how-does-gguf-work

That’s it for now.