How Prefill, Decode, HyDE, and Hybrid Search Work

Seven new blogs this week, covering LLM inference internals, three ways to make retrieval and caching smarter, PyTorch fundamentals, and two prompting techniques.

Let’s get started.

Every time an LLM answers a request, it goes through two distinct phases.

Prefill is the phase where the model reads and processes your entire input prompt in one single pass and produces the very first output token. Decode is the phase where the model generates the output tokens one at a time, reusing the KV cache that prefill prepared.

In simple words, prefill is the model reading your question, and decode is the model writing the answer.

Because the two phases have opposite bottlenecks, each optimization targets one of them. Prefill is compute-bound and sets TTFT, which is how fast the answer starts. Decode is memory-bandwidth-bound and sets TPOT, which is how fast the answer streams.

Read here: https://outcomeschool.com/blog/prefill-vs-decode-llm-inference-optimization

HyDE stands for Hypothetical Document Embeddings. It is a technique where we first ask the AI model to write a fake answer to the question, and then we search using that fake answer instead of the question.

The core idea is one beautiful trick. An answer looks like an answer. So, if we want to find a real answer, we must search using something that looks like an answer. The question does not look like an answer. But a fake answer does look like an answer.

This is the heart of HyDE. We search with a fake answer to find the real answer.

Read here: https://outcomeschool.com/blog/how-does-hyde-work

Hybrid Search is a technique that combines keyword search and semantic search, and merges their results into one final ranked list.

Keyword search is great at exact words and codes, but blind to meaning. Semantic search is great at meaning, but weak at exact codes and rare terms. Each one is strong exactly where the other one is weak.

In simple words, Hybrid Search runs both searches at the same time, and then combines what they found. This way, we get the exact-match power of keyword search and the meaning power of semantic search, together in one result.

Read here: https://outcomeschool.com/blog/how-does-hybrid-search-work

Semantic Caching is a cache that matches questions by their meaning instead of their exact words.

In simple words, if a new question means the same thing as an old question, we return the old answer, even if the words are different.

Going back to our example, “What is the capital of France?” and “Tell me the capital city of France.” mean the same thing. So, Semantic Caching treats them as a match and returns “Paris” without asking the model again. This is the beauty of Semantic Caching. It understands intent, not just text.

Read here: https://outcomeschool.com/blog/how-does-semantic-caching-work

PyTorch is a free and open-source library that helps us build and train machine learning models.

In simple words, PyTorch is a toolbox. It gives us ready-made tools so that we do not have to build everything from scratch.

To do the hard math for us automatically, PyTorch uses two powerful ideas. The first is the Computation Graph, which is a record of every math step that PyTorch performs, kept in the right order. The second is Autograd, which automatically calculates how to adjust each number to reduce the error. We write the forward calculation in plain code, and PyTorch handles all the backward math.

Read here: https://outcomeschool.com/blog/how-does-pytorch-work

Prompt Chaining is a way of breaking one big task into smaller prompts, where the output of one prompt becomes the input of the next prompt.

In simple words, instead of asking the AI to do everything in one single big prompt, we ask it in small steps. We take the answer from the first step and feed it into the second step, and so on.

Each step does only one small job. If something goes wrong, we know exactly which step to fix. This is why we need Prompt Chaining. It makes a hard task easy by splitting it into small, simple steps.

Read here: https://outcomeschool.com/blog/how-does-prompt-chaining-work

Chain-of-Thought (CoT) Prompting is a technique where we ask the model to write out its reasoning steps before giving the final answer.

In simple words, instead of asking for the answer directly, we ask the model to “think out loud” first.

When the model writes out its reasoning, those reasoning words become part of what it reads next. So the model is now building its final answer on top of correct, visible steps, instead of guessing from nothing. So, the model is not getting smarter. We are just giving it room to think, and that too in a structured way.

Read here: https://outcomeschool.com/blog/how-does-chain-of-thought-prompting-work

That’s it for now.