Lesson 3: Embeddings — Turning Text Into Searchable Vectors

Today we build:

• A deterministic text embedder — no API call, no training, no cost

• A 256-dimension vector for every passage from Day 2

• The similarity search that Day 4’s retrieval stage will build on — including an honest look at where it falls short

Yesterday’s passages are just text — comparing a question against them character by character finds nothing unless the words match exactly. To search by meaning, text has to become numbers first: a vector, positioned so that similar meaning means similar position. Today we build the simplest version of that idea that still works, using a technique real production systems actually use — not a toy simplification of one.

The hashing trick turns each word in a passage into a position in a fixed-length vector, using a hash function instead of a learned vocabulary. Every occurrence of a word nudges the same vector position by the same amount, so two passages that share vocabulary end up with vectors pointing in a similar direction — measurable with cosine similarity, a score from -1 (opposite) to 1 (identical).

This is a real technique, used at scale specifically because it needs no training step and no vocabulary file — you can embed text the moment you see it. It’s also genuinely weaker than a learned embedding model, and today’s run shows exactly why: with only a handful of passages and short text, shared common words and hash collisions can outweigh the one word that actually matters for relevance. Stripping obvious stopwords (”the,” “a,” “can”) helps, but doesn’t eliminate the problem — it’s a structural limit of the technique, not a bug in today’s code.

Text goes in, a fixed-length vector comes out — deterministically, so the same passage always produces the same vector. Embedder.embed_passages() runs this over every passage from Day 2’s passage store and writes the result to embeddings.json. rank_by_similarity() previews what Day 4’s retrieval stage will do: embed a query the same way, then rank passages by cosine similarity.

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_03_package/lesson_03_package

Two properties are worth testing directly, because they’re provably true regardless of vocabulary quirks: identical text always produces similarity 1.0, and text with zero shared vocabulary always produces similarity 0.0.

Expected output:

Embedding 4 passages (dim=256)...
  doc_refund_p0 -> vector with 9 non-zero dims
  doc_refund_p1 -> vector with 10 non-zero dims
  doc_cancel_p0 -> vector with 7 non-zero dims
  doc_pricing_p0 -> vector with 10 non-zero dims
Embeddings saved -> embeddings.json
Query: "Can I get my money back on an annual subscription?"
  doc_cancel_p0  similarity=0.169
  doc_refund_p0  similarity=0.149
  doc_pricing_p0  similarity=0.124
== Running tests ==
6 passed

Notice the top result, doc_cancel_p0, isn’t obviously the best answer to a refund question — doc_refund_p0 (the actual refund policy) ranks second. That’s not an error; it’s the hashing trick’s real behavior on a small, short passage set, and it’s genuine motivation for Day 4.

lesson_03_package/
├── lesson_code.py    # tokenize, embed_text, cosine_similarity, Embedder
├── test_lesson.py    # 6 tests verifying embedding correctness properties
├── requirements.txt  # pytest only — no API keys needed
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, embeddings.json, caches, Docker image
└── README.md          # quick-start reference

You should see 4 passages embedded into 256-dimension vectors, an embeddings.json file written, a similarity ranking for a sample query, and 6 passed.

When you’re done: ./stop.sh.

embed_text() is the core function — everything else calls it:

1. tokenize() lowercases the text, strips punctuation, splits on whitespace, and drops stopwords.

2. For each remaining token, MD5-hash it, use the hash to pick a vector index (hash % dim) and a sign (+1 or -1, from a different bit of the same hash).

3. Add that signed value into the vector at that index.

4. L2-normalize the final vector — divide every value by the vector’s magnitude — so cosine similarity is comparing direction, not raw word count.

Embedder.embed_passages() runs this over a list of passages and returns one PassageEmbedding per passage. rank_by_similarity() embeds a query with the same function and sorts passages by cosine_similarity() against it.

Running this lesson produces:

doc_cancel_p0  similarity=0.169
doc_refund_p0  similarity=0.149
doc_pricing_p0  similarity=0.124

for the query “Can I get my money back on an annual subscription?” — even though doc_refund_p0 is the passage that actually answers a refund question. This is real, reproducible behavior, not a bug to fix quietly. Two things are happening:

• With only 4 passages of 7–11 tokens each, a single shared word (annual, subscription) carries a lot of weight in either direction — there isn’t enough text for chance overlaps to average out.

• MD5-hashing into a 256-dimension space still has occasional collisions between unrelated tokens, and at this passage count, one lucky or unlucky collision can flip the ranking.

Both effects shrink as passage count and text length grow, and disappear entirely with a learned embedding model (which this lesson deliberately doesn’t use yet, to stay free and dependency-light). The two tests that matter for correctness — identical text scores 1.0, completely disjoint vocabulary scores 0.0 — hold exactly, because those are mathematical properties of cosine similarity and L2 normalization, not properties that depend on passage count.

• Different similarity scores than the article — if you change MOCK_PASSAGES, EMBEDDING_DIM, or STOPWORDS, expect different rankings; the hashing trick is deterministic per configuration, not across configurations.

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_03_package.

• Wondering why we didn’t just use a real embedding API today — that’s intentional. Day 3 stays free and dependency-light on purpose; later phases introduce real API-backed embeddings once the eval harness (Phase 1) exists to catch regressions when you do.

This is exactly why production retrieval systems rarely trust a single vector-similarity score by itself. A system that only ranks by embedding similarity will occasionally surface a plausible-looking wrong answer with high confidence — and nothing in the score itself signals the mistake. That gap is a large part of why hybrid approaches exist.

Day 4, we add hybrid search: combining today’s vector similarity with keyword-based scoring, so a passage that shares the exact right word ranks correctly even when the embedding alone gets it wrong.