Full-text search (BM25)

Full-text search breaks documents into normalized terms, stores them in an inverted index, and finds documents containing the query's terms. Ranking functions decide order; BM25, the most widely used, rewards documents where query terms are frequent, gives more weight to rare terms, and adjusts for document length so long documents don't win just by being long.

In AI products it's the complement to vector search. Keywords excel exactly where embeddings struggle: product codes, error messages, invoice numbers, people's names, and exact phrases. They are fast, cheap, explainable, and need no model calls at query time.

Postgres has built-in full-text search using tsvector columns, tsquery queries, and a GIN index, with ranking functions such as ts_rank_cd. Those built-in rankers are not BM25, though they work well as one half of a hybrid; extensions such as ParadeDB's pg_search add true BM25 if you need it. Dedicated search engines implement BM25 natively.

Watch tokenization and language settings: stemming helps refunds match refund but can mangle identifiers, and multilingual content needs the right configuration per language. Check how your specific IDs and codes are tokenized before trusting results.