Embeddings & Vector Search· Lesson 7 of 7
Retrieval Quality — Measuring What You Get Back
In one line · Retrieval quality measures whether the chunks you got back actually contain the answer — because the model can't answer well from the wrong context.
Most RAG problems aren't in the model, they're in retrieval: feed it missing or wrong context and it hallucinates. If you don't measure, you can't fix.
Two simple metrics: recall (did you fetch all the relevant chunks?) and precision (how much of what you fetched was actually on-topic?). Together they show your search's health.
Like an assistant you send to fetch files on a topic: recall asks whether they brought every relevant file or missed some, and precision asks how much of the pile was actually on-topic versus junk. You want both high.
Take 20 questions with known answers and check how often retrieval puts the right chunk in the top 3; if it's only 12/20, you know chunking or the model needs work.
You build a small evaluation set: 20-50 questions, each mapped to the chunk (or chunks) known to contain its answer — the ground truth. Run retrieval on every question and compute: recall@k (what fraction of the right chunks appeared in the top k) and precision@k (what fraction of the top k was actually right). There's a natural tension: raising k lifts recall (better odds of catching the right chunk) but lowers precision (more filler rides along). Once the metric is in place, every change — chunk size, model, hybrid, reranking — gets measured in numbers before and after, not by feel.
🧪 Try it: write 20 questions over your documents and mark which chunk holds each answer. Compute recall@3 and precision@3 for the current system and record both numbers. Then change exactly one thing (chunk size, say) and re-measure — now a number tells you whether the change helped or hurt, and that's the difference between engineering and guessing.
The biggest mistake: evaluating by feel — 'I tried 3 questions and it seemed fine' — which hides quality regressions whenever you change something. Second: fixing low recall by just raising k — it can pass, but you stuff the context with irrelevant chunks that distract the model and inflate cost. Third: judging the whole RAG from the final answer alone — measure retrieval in isolation first, because a wrong answer can come from good retrieval plus a bad generation, or the reverse.
Build an evaluation set as soon as your RAG has real users or decisions ride on its answers, and before any big change (new model, new chunking) so you can compare before/after in numbers.
In the first week of prototyping, quick manual inspection is enough — don't delay an experimental launch to build a full evaluation harness. But the moment you start tuning and optimizing, measurement becomes mandatory, not a luxury.