Chunking Strategies That Work

You can retrieve only whole chunks, so the way you split documents decides what retrieval can ever return. Split too large and a chunk about five topics matches none of them well. Split too small and you get the chunk that says "This is extended to 60 days for annual plans" with no hint of what is extended. Split carelessly and a pricing table ends up half in one chunk and half in the next.

Chunking gets debated more than almost anything else in retrieval, and most of the debate isn't settled by opinion. This lesson compares the three families (fixed-size, structural, and semantic) with working code, gives you defensible starting values for size and overlap, and then covers the technique that usually matters more than the splitter: contextual chunk headers that make every chunk understandable on its own.

What a chunk has to do#

A chunk does two jobs, and they pull in different directions:

1. It's the unit of matching. Its embedding (and its keywords) must be specific enough that the right query finds it. Smaller, focused chunks match more precisely.
2. It's the unit of context. When retrieved, it goes into a prompt or onto a results page and must carry enough information to be useful. Larger chunks carry more context.

Every chunking decision trades these off. Two failure modes show up in practice:

• Diluted chunks: a chunk covering several topics produces an embedding that's the average of all of them and matches none strongly. Long FAQ pages chunked as one block are the classic case.
• Orphaned chunks: a chunk that depends on text it doesn't contain. Pronouns ("it," "this plan"), section-relative statements ("the limit above"), and table rows without their header row all produce chunks that look irrelevant to the query they actually answer.

Good chunking avoids dilution by keeping chunks focused, and contextual headers (later in this lesson) fix orphaning without making chunks bigger.