Chunking

Embedding a whole long document produces one vector that blurs every topic in it, and retrieving it fills the prompt with mostly irrelevant text. Chunking breaks documents into passages, typically a few hundred tokens each, that are embedded and retrieved individually.

There are three broad families. Fixed-size chunking splits by token count, often with some overlap between neighbors. Structural chunking follows the document's own shape, such as headings, sections, and list boundaries. Semantic chunking looks for topic shifts. Structural chunking is often the best starting point for well-formatted content like help articles and docs.

Two failure modes guide the choice. Chunks that are too big dilute: the relevant sentence is buried among unrelated ones, which weakens the embedding. Chunks that are too small orphan: a passage that says it applies to annual plans loses the context of which product or policy it belongs to. Adding a short contextual header, such as document title and section path, to each chunk often fixes orphaning more effectively than changing the splitter.

Don't pick settings by opinion. Try two or three configurations on a labeled query set and keep the one with better recall.