Lesson 2: Ingestion — Getting Real Data Into the System

Today we build:

• A document ingestor that turns raw text into uniform, retrievable passages

• A fixed-size chunking strategy with overlap, so context doesn’t get cut mid-thought

• The passage store that Day 4’s retrieval stage will search over

Yesterday’s RetrievalStage was a hardcoded dictionary — real answers to three fake questions. Today it becomes real, because real documents don’t arrive in question-answer pairs. They arrive as long, unstructured text: policy docs, support articles, transcripts. Before any search can work, that raw text has to become a set of small, consistent units a retrieval system can actually compare against a query. Get chunking wrong here, and every later phase inherits the mistake — no eval harness or LLM judge can fix an answer built from a badly cut passage.

A document is too big to search directly. If you embed and compare a 4,000-word support article against a 12-word question, the match is diluted — the one relevant sentence gets buried in three thousand irrelevant ones. So we split documents into passages: smaller chunks a retrieval system can score individually.

The chunking decision has a real trade-off. Chunks too large re-introduce the dilution problem. Chunks too small lose context — a passage that ends mid-sentence, or separates a term from its definition, can’t be understood on its own even if it’s retrieved correctly. Today we use fixed-size character chunking with overlap: each passage shares a small window of text with its neighbor, so an idea that spans a chunk boundary still appears whole in at least one passage.

Every passage also carries metadata: which document it came from, and where in that document it sits. This isn’t optional bookkeeping — Day 4’s retrieval stage needs it to return results a user can trust, and Day 3’s embeddings need a stable ID to attach a vector to.

Ingestion sits before the pipeline, not inside it: it’s a one-time (or periodically re-run) process that produces the passage store Stage 2 (Retrieval) will query. Raw documents go in one end; a flat list of Passage objects — each with an ID, source document, character offsets, and text — comes out the other.

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_02_package/lesson_02_package

Two data contracts, matching the discipline from Day 1:

The ingestor does one job per method: load_documents() reads raw text in, chunk_document() splits one document into overlapping passages, and ingest_all() runs both across every document and returns the full passage store.

Expected output:

Loaded 3 documents
Document 'refund_policy.txt' -> 3 passages
Document 'cancellation.txt' -> 1 passages
Document 'pricing.txt' -> 1 passages
Total passages ingested: 5
Passage store saved -> passages.json
== Running tests ==
5 passed

lesson_02_package/
├── lesson_code.py    # Document, Passage contracts + DocumentIngestor
├── test_lesson.py    # 5 tests verifying chunking behavior
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, passages.json, caches, Docker image
└── README.md          # quick-start reference

start.sh creates a virtual environment, installs requirements.txt, builds the Docker image if Docker is present, runs lesson_code.py, then runs test_lesson.py. You should see 3 documents ingested into 5 total passages, a passages.json file written to disk, and 5 passed.

When you’re done: ./stop.sh — removes the venv, the generated passages.json, all caches, and the Docker image if one was built.

Two dataclasses carry the data:

Type Fields Produced by Document doc_id, source, raw_text load_documents() Passage passage_id, doc_id, text, char_start, char_end chunk_document()

DocumentIngestor.ingest_all() is the entry point: it calls load_documents() once, then chunk_document() once per document, and returns a flat list of Passage objects — the passage store.

• chunk_size=220 characters keeps each passage small enough that a single topic dominates it, without cutting it down to a fragment.

• overlap=40 characters means each new chunk starts 40 characters before the previous one ended. If a sentence spans the boundary between chunk N and chunk N+1, it appears complete in chunk N+1 even though it was cut off in chunk N.

• Short documents (shorter than chunk_size) produce exactly one passage — there’s nothing to split, and the loop exits after the first pass since end == len(text).

This is deliberately the simplest correct chunking strategy. Production systems often chunk on sentence or paragraph boundaries instead of raw character counts — that’s a refinement worth making later, but it changes chunk_document()‘s internals only. The Passage contract and everything downstream stays the same, which is the entire point of the typed-boundary discipline from Day 1.

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside the lesson_02_package directory.

• Different passage counts than expected — passage counts depend on the exact chunk_size and overlap values in DocumentIngestor(); if you change them, the counts in the article and README will no longer match your run — that’s expected, not a bug.

• passages.json already exists from a previous run — start.sh overwrites it each run; stop.sh removes it during cleanup.

This is the step most teams skip past without thinking about — and the one that quietly determines whether retrieval works at all. A support-triage system that chunks a refund policy document badly will confidently retrieve a passage that’s missing the one sentence that mattered, and no amount of prompt tuning downstream fixes that. Chunking strategy is infrastructure, not a detail.

Day 3, we turn these passages into embeddings — numeric vectors that let the retrieval stage compare a question against thousands of passages by meaning, not just keyword overlap.