Design the Exam

Your RAG assistant is live and people are using it — but nobody can say how good it is. The only verdict available right now is 'we tried it and it seemed fine'. You have the documents it answers from, a week of real user questions, and the KB failure-mode catalog: questions the docs answer, questions they don't answer at all, edge questions (answerable but partial or split across two documents), and adversarial questions designed to bait it. Build a 12-case exam that becomes the judge before every ship.

Success: A 12-case golden set covering all four categories, each with a judgeable expected behavior.