The Broken Eval

A team's RAG assistant "passes its eval" at 96% — yet users report wrong answers daily. The eval itself is broken: the golden set was generated by the same model being evaluated, the questions are trivially easy and leak the answers, faithfulness is measured by string overlap, and the whole test set is 12 questions. Diagnose why the high score lies, then fix the eval itself.

Success: Diagnose all four eval flaws and build a new eval that catches the errors users actually see — without overfitting to it.