Measuring LLM quality: evals and RAG
How to tell whether your LLM app is any good and how to make retrieval find the right thing, in watch order. Hand-drawn doodle explainers, theory then a practical example.
1. How LLM Evals Work
Vibes are not a test suite. Build your first LLM eval: a dataset from real failures, your real app, and a grader you can trust, wired into CI.
2. RAG That Actually Retrieves the Right Thing
Most bad RAG answers start in search. Fix retrieval with structure-aware chunks, hybrid keyword plus vector search, a reranker, and a recall test you can rerun.
play with it
Is that eval win real?
Two prompts, one eval set. Is the better score real, or luck of the cases?
A: 82% (likely between 69% and 90%)
B: 86% (likely between 74% and 93%)
The ranges overlap: not a clear win yet. Add cases or look at which ones changed.