Long context

With context windows reaching around a million tokens on several current models, you can often send an entire manual, contract set, or codebase in one request. Long context lets the model see everything at once, which helps with questions that need connections across many parts of a document and removes the retrieval step that can miss the right passage.

The decision is mostly arithmetic. Every token is billed on every call, so sending a large corpus with each question can be expensive and slow. Prompt caching changes the math considerably: if the same documents are reused across many requests, cache reads cost a small fraction of normal input and cut latency too.

Long context has real limits. Models don't attend equally well to everything in a very long input, so details can be overlooked, and research such as the Lost in the Middle paper found weaker use of information placed mid-context in the models it tested. Corpora that grow, change often, or need per-user permissions still favor retrieval. And time to first token rises with input size.

A practical middle path is retrieving whole relevant documents rather than small chunks, then letting long context handle the reading.