takeaway notes
RAG That Actually Retrieves the Right Thing
After this you can debug a RAG feature from the search side. You print and log the chunks each answer used, so a wrong answer can be traced to a missing chunk. You split documents on their structure with a little overlap and label each chunk with its page and section, run keyword and vector search together, rerank the top twenty with a cross-encoder, and measure recall at k on twenty real questions every time you change something.
Concepts
What RAG is
Theory
Retrieval-augmented generation has three steps: search your documents for passages that match the question, paste them into the prompt, and let the model answer from them. It is an open-book exam where your search picks the pages. If all your documents fit in the prompt, you may not need retrieval at all.
Practical example: Is uploading a PDF RAG?
Only if something searches the PDF and picks pieces. Putting the whole PDF in the prompt is a long prompt, which is fine for small documents.
Debug search first
Theory
The model can only use the chunks you give it, so check retrieval before you touch the prompt. Position matters too: models often use facts at the start or end of a long context better than facts in the middle, so more chunks is not always better.
Practical example: Log what the model saw
Store the ids of the retrieved chunks with every answer, so a complaint can be replayed against exactly the context the model had.
Chunking
Theory
Documents are cut into chunks, usually a few hundred tokens each. Cutting at a fixed character count can split one rule across two chunks, and the second half may not mention the topic at all.
Practical example: The refund rule cut in half
Chunk one: 'Refunds are available within 30 days.' Chunk two: 'After that, annual plans are prorated for unused months.' Split on headings and paragraphs, overlap by a sentence or two, and prepend a label such as 'Billing > Refunds' to every chunk.
Embeddings and vector search
Theory
An embedding model turns text into a vector; texts with similar meaning land close together. Search embeds the question and returns the nearest chunks, often by cosine similarity. Approximate indexes such as HNSW trade some recall for speed.
Practical example: Check your index
In Postgres with pgvector, a query without an index is exact. Run a sample of questions both ways and compare the results.
SELECT id FROM chunks ORDER BY embedding <=> $1 LIMIT 5;Hybrid search
Theory
Embeddings are good at meaning and weak at exact strings such as error codes and product names, where keyword search (BM25) shines. Hybrid search runs both and merges the two ranked lists with reciprocal rank fusion, which needs no score tuning.
Practical example: Reciprocal rank fusion
Each list gives a chunk 1 / (60 + rank). A chunk ranked first in both lists scores 2/61, about 0.033, and beats a chunk that is first in only one list (1/61, about 0.016).
def rrf(lists, k=60):
scores = {}
for ranked in lists:
for rank, doc in enumerate(ranked, start=1):
scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
print(rrf([['a', 'b', 'c'], ['a', 'c', 'd']])) # ['a', 'c', 'b', 'd']Reranking
Theory
A fast first search is built for scale and can return junk. A cross-encoder reranker reads the question and each candidate together and scores relevance; it is more accurate but too slow for a whole library, so it only reorders a short list.
Practical example: Retrieve twenty, keep five
Retrieve the top twenty with hybrid search, score each with a cross-encoder, sort, and put the best five in the prompt.
Measure retrieval
Theory
Recall at k is the share of test questions whose right chunk appears in the top k results. Measure it on its own, separately from answer quality, and rerun it after every change.
Practical example: Twenty questions
Write twenty real user questions and mark the chunk that answers each. If fourteen show up in the top five, recall at five is 70 percent. Keep every question that ever failed in the set.
Checklist
- Print the retrieved chunks for one wrong answer and check whether the answer is in them.
- Log retrieved chunk ids with every answer.
- Re-chunk on headings and paragraphs with a sentence or two of overlap.
- Prepend each chunk's page and section title before embedding and indexing it.
- Add keyword search next to vector search and merge with reciprocal rank fusion.
- Rerank the top twenty candidates and keep the best five.
- Write twenty labelled questions and track recall at five on every change.
Gotchas
- Tuning the prompt when the answer was never retrieved: check the chunks first.
- Stuffing in more chunks to be safe: the one that matters can get lost in the middle.
- Fixed-size chunks that split a rule in two: split on structure and overlap.
- Relying on vectors alone for codes, names and SKUs: add keyword search.
- Mixing embedding models between your chunks and your questions: their vectors don't compare.