Capstone 2: A RAG Knowledge Base App
Build a cited Q&A app over a document collection: ingestion, hybrid search, reranking, grounded answers, permission filters, and a retrieval eval suite.
Every company has the same problem: the answer exists, somewhere, in a handbook page, a runbook, or a help article nobody can find. A knowledge base app that answers questions from those documents, with a citation on every claim, is one of the most requested AI features there is. It's also one of the easiest to get subtly wrong: answers that sound right but cite the wrong page, a retriever that can't find an error code, or worse, an answer built from a document the person asking was never allowed to see.
In this capstone you'll build the whole thing for the Deskmate team: ingestion that re-runs safely, hybrid search, reranking, grounded answers with verifiable citations, permission filters enforced in the database, and an eval suite that measures retrieval separately from generation.
What you'll build
An internal Q&A app where Deskmate employees ask questions like "What's our refund window for annual plans?" or "How do I rotate the SSO certificate for a customer?" and get a short answer with numbered citations. Click a citation and you see the exact passage it came from. Ask something the documents don't cover and the app says so instead of guessing.
The document collection has three audiences, which is what makes permissions real:
| Folder | Example content | Who can read it |
|---|---|---|
docs/everyone/ | Employee handbook, public help articles | All employees |
docs/support/ | Support runbooks, escalation procedures | Support team |
docs/finance/ | Refund approval limits, billing exceptions | Finance team |
INGEST (script, re-runnable) QUERY (per question)
docs/<group>/*.md POST /api/ask (user from session)
│ hash unchanged? → skip │
▼ ▼
chunkMarkdown() by headings rewriteQuery() Haiku, only if needed
│ + contextual header │
▼ ▼
embed() ──▶ chunks table hybridSearch() pgvector + full-text, RRF
(vector, tsvector, │ permission filter in SQL
allowed_groups) ▼
rerank() Haiku scores top 30, keeps 6
│ none relevant? → "not covered"
▼
answer() Sonnet + search_result blocks
│ citations from the API
▼
segments + numbered sourcesEach box maps to a lesson in Module 5, and the capstone links to them as you go instead of re-teaching them. The new part is putting them together so the seams hold.