Ingesting Documents: PDFs, HTML, and Office Files
Turn real-world files into clean, structured text: extraction tools, tables and layout, metadata worth keeping, and an ingestion pipeline you can re-run.
Every retrieval demo starts with clean Markdown. Real customers upload a 90-page PDF exported from a slide deck, a Word document with headings that are just bold text, a help center scraped as HTML with navigation menus on every page, and a scanned contract with no text layer at all. Feed that into your embedder as-is and search quality is capped before you write a single query: the right answer may be in the file, but it never made it into your index in a usable form.
This lesson is about the unglamorous stage that decides retrieval quality: ingestion, turning real files into clean, structured text with the metadata you'll need later. You'll build extractors for HTML, Word, and PDF, learn when to reach for heavier tools or a vision model, decide what metadata to keep, and wrap it all in a pipeline you can re-run whenever your extraction improves.
Why ingestion caps retrieval quality
Retrieval can only find what ingestion produced. The failures are rarely dramatic errors; they're quiet corruptions that make chunks less findable and answers less accurate:
- Boilerplate: navigation menus, cookie banners, and footers repeated on every page. They dilute embeddings and make unrelated pages look similar.
- Broken reading order: two-column PDFs extracted line by line across both columns, interleaving unrelated sentences.
- Flattened tables: a pricing table becomes "Plan Agents Starter 3 Pro 50", and no reader, human or model, can tell which number belongs to which plan.
- Lost structure: headings turned into ordinary text, so you can't chunk by section or tell the model where a passage came from.
- Missing text: scanned pages and images of text with no text layer, which extract as empty strings and silently vanish from search.
- Artifacts: words hyphenated across line breaks ("refund-\nable"), running headers ("Page 3 of 41") inside paragraphs, and odd characters from fonts.
None of these throws an error. The only way to catch them is to look at extracted output, which is why the pipeline below records warnings and keeps its output inspectable.