INCIDENT BOARD · EVALUATION
Challenges
Real failure modes on real systems. The clock runs; the evaluation decides.
TODAY'S CHALLENGESEV-1 · OPEN⭐⭐⭐ · 250 PTS · 90 min
RAG Hallucination HuntA RAG system gives confident wrong answers even though the source exists. You have traces, the corpus, and 90 minutes. Fix it without tipping into over-refusal.Open incidentSCORED ONCorrectness · 40%Approach · 25%Cost · 20%Robustness · 15%Result: a gate + the dimensions this challenge measures
SEV-3Early draft⭐ · 30 min
Spot the HallucinationA support bot gives confident answers with invented citations and specs. You have the traces and time. Diagnose and fix.LLMs · after the hallucination conceptStart →
SEV-3Early draft⭐ · 40 min
Context OverflowA chat answers well for 10 turns then forgets or errors. Diagnose and fix without blowing up the cost.LLMs → AI Chat AppStart →
SEV-2Early draft⭐⭐ · 60 min
Break Your Own BotAn adversarial message script tries to make your bot drop its persona and leak its instructions. Harden it.Prompt Eng → Support AgentStart →
SEV-1Early draft⭐⭐⭐ · 90 min
The $40 Agent RunThe research agent finishes its work correctly and terminates on its own — and still costs $40 a run. The trace shows just 12 steps, no looping or repetition: every step resends the full text of every page read so far, and all 12 run on the most expensive model — including 'pick the next link'. This cost is architecture, not a bug. Cut it to ≤ $4 at the same report quality on the eval.Research Agent · final runStart →
SEV-1Early draft⭐⭐⭐ · 90 min
The Broken EvalA team's RAG assistant "passes its eval" at 96% — yet users report wrong answers daily. The eval itself is broken: the golden set was generated by the same model being evaluated, the questions are trivially easy and leak the answers, faithfulness is measured by string overlap, and the whole test set is 12 questions. Diagnose why the high score lies, then fix the eval itself.Evaluation · after the evaluator-optimizer patternStart →
SEV-2Early draft⭐⭐ · 45 min
The Retry StormRetries pile up, causing extra load and inflated bills.AI APIs · error handlingStart →
SEV-2Early draft⭐⭐ · 50 min
Drama-Free ExtractionStructured extraction sometimes returns broken JSON.AI APIs · structured outputsStart →
SEV-2Early draft⭐⭐ · 45 min
Retrieval Returning GarbageRetrieval returns irrelevant results.EmbeddingsStart →
SEV-2Early draft⭐⭐ · 60 min
Retrieval AutopsyPoor retrieval breaks answers.RAG · retrieval stageStart →
SEV-3Early draft⭐ · 45 min
The Expensive PromptA prompt costs more than it should per call.LLMsStart →
SEV-2Early draft⭐⭐ · 50 min
The Prompt That Broke in ProductionA prompt worked, then started returning wrong results. A 'tiny' tweak to the classification prompt shipped Friday with zero testing — Monday morning the complaints started: tickets misclassified in production. You have the git history, the traces, and the user reports.Prompt EngStart →
SEV-1Early draft⭐⭐⭐ · 60 min
The RAG Cost BlowupRAG cost ballooned for no clear reason. The bill jumped 8x this month. A quick look shows someone 'improved recall' by raising k from 3 to 20 chunks per query. Quality nudged up slightly — cost exploded. You have the traces, the billing numbers, and an eval set.RAG · cost stageStart →
SEV-2Early draft⭐⭐ · 50 min
The Slow AgentThe agent over-calls tools, going slow and costly. Traces show it firing search_kb and lookup_order even for questions like 'how do I change my password?' — answered in the system prompt itself. Every call adds a round-trip; latency is 3x and cost follows.Support Agent · optimizationStart →
SEV-2Early draft⭐⭐ · 50 min
Tool Failure CascadeA tool outage cascades through the agent. The order-lookup tool went down for 20 minutes today — and instead of coping, every conversation touching orders ended in an exception or a raw error dumped on the user; even order-unrelated questions got hit. You have the code and the traces.Support Agent · tool stageStart →
SEV-1Early draft⭐⭐⭐ · 60 min
The Agent That Never StopsThe agent sometimes loops forever. Yesterday one research run on a vague goal ('find the best pricing strategy') spun for two hours and burned $40 before someone noticed and killed it by hand. The trace shows the same searches repeating in different phrasings, endlessly. You have the code and the traces.Research Agent · reliabilityStart →
SEV-3Early draft⭐ · 40 min
The Right ModelThree products on your desk, each needing a model decision today. (1) Support-ticket classification: 10M tickets/month, two labels only — complaint or inquiry — on a $200/month budget. (2) In-product assistant chat: the user must see the first word in under a second; questions are ordinary and mid-length. (3) Legal contract analysis: 40 contracts/month, ~60 pages each, where one missed clause can cost the company many times the entire bill. You have a model landscape sheet: small (cheap, fast), mid, and frontier (far pricier, sharper on hard reasoning).LLMs · model selectionStart →
SEV-2Early draft⭐⭐ · 60 min
Design the ExamYour RAG assistant is live and people are using it — but nobody can say how good it is. The only verdict available right now is 'we tried it and it seemed fine'. You have the documents it answers from, a week of real user questions, and the KB failure-mode catalog: questions the docs answer, questions they don't answer at all, edge questions (answerable but partial or split across two documents), and adversarial questions designed to bait it. Build a 12-case exam that becomes the judge before every ship.RAG Knowledge Assistant · evaluation stageStart →
SEV-2Early draft⭐⭐ · 50 min
Which One Is Right?Three cards on the table, each needing an architecture decision: RAG, long context, fine-tuning, or no LLM at all. (1) An insurance knowledge bot: 4,000 policy documents, edited weekly, and every answer must point at its source because a regulator reviews them. (2) A marketing copy generator: the knowledge required is trivial, but the tone must land in the brand's exact voice every time, and you have 3,000 past pieces written in that voice. (3) An FAQ page: 40 fixed question-answer pairs unchanged for a year, 200 queries a day.RAG · domain capstoneStart →
HOW SCORING WORKSEvery attempt is scored on the same dimensions — correctness, approach, cost, robustness. Best score counts, attempts are unlimited, and a pass earns 'evaluated' skill signals.