The Evaluator–Optimizer Pattern — One Builds, One Judges

Two parties with separate roles. The first one builds. The second holds a criteria sheet — not personal taste — and judges the work item by item: pass / fail.

If an item fails, specific feedback goes back to the first: "You failed 2 and 3, and here's why." The first redoes the work knowing exactly what to fix. And so on until the work passes every item — or we hit the attempt cap, at which point we stop and send the output for human review instead of looping forever.

An exam grader with a model answer and a scoring scheme: "correct definition = 2 points, valid example = 1 point." They don't grade by mood — item by item. And the student who got 3 out of 5 knows exactly what they got wrong, so their next study session is targeted.

Compare that with a grader who has no answer key, reads through, and writes "very good" by feel — that's the Reflection critic without a rubric.

And there's a third, worse possibility: a grader holding the wrong answer key — confidently awarding high marks to bad answers. That is exactly the subject of the "The wrong exam" challenge — because the most dangerous thing in this pattern isn't the absence of a judge... it's a judge that falsely reassures you.

A real example

The task: landing copy for an expense-management app for small businesses. The Evaluator holds 4 named criteria:

> - C1: The core benefit is clear in the first sentence > - C2: Exactly one CTA > - C3: No numbers or claims that aren't in the brief > - C4: Length ≤ 80 words

Attempt 1 — the Generator produced copy containing "save 70% of your accounting time" and closed with both "Try it free" and "Request a demo":

Evaluator report (attempt 1):
  C1: PASS — benefit stated in the first sentence
  C2: FAIL — two CTAs found: "Try it free" and "Request a demo"
  C3: FAIL — the 70% claim does not exist in the brief
  C4: PASS — 64 words
  verdict: REJECT → feedback sent back to generator

Attempt 2 — the Generator removed the number and consolidated on a single "Try it free" CTA:

Evaluator report (attempt 2):
  C1: PASS   C2: PASS   C3: PASS   C4: PASS
  verdict: ACCEPT → ship it

Notice two things: the feedback was specific to the item ("the number isn't in the brief"), not a generic score — which is why the second attempt fixed exactly what was needed. And the criteria were named and written before the run — not an opinion invented at judgment time.

1. Generate: the Optimizer produces an output — with any feedback from the previous round included in its prompt. 2. Evaluate: the Evaluator checks the output against every criterion and produces a structured report: PASS/FAIL per item + the reason. 3. Decide: all PASS? The output ships. Any FAIL with attempts remaining? The feedback goes back to step 1. 4. The exit path: attempts exhausted and still failing? The output is flagged for human review — it does not keep looping.

### So where do the criteria themselves come from?

This is the most important part of the whole pattern — and here's its ladder of strength:

The rule: automate whatever can be automated with code, and leave the LLM judge only the criteria that genuinely require language understanding. A length criterion, for instance (C4 above), has no business being judged by an LLM — one line of Python settles it for free.

### A frank word about LLM-as-Judge

• Self-preference is real: a model tends to score text that resembles its own style higher — which is why judge model ≠ generator model is the healthier practice.
• Documented biases: it favors longer answers, and it's sensitive to presentation order in comparisons.
• Calibration is a duty, not a luxury: measure its judgments against a human-graded sample (even 30 examples make a difference). If its agreement with humans is weak — you have a wrong exam, and every score after that is meaningless.

The whole loop, with any provider — note where the decisions live:

MAX_ATTEMPTS = 3                              # always cap!
feedback = None

for attempt in range(MAX_ATTEMPTS):
    output = generator(task, brief, feedback)        # optimizer produces
    report = evaluator(output, criteria)             # judge: per-criterion
    if report.all_passed:                            # programmatic decision
        return output                                # quality gate passed
    feedback = report.failures                       # specific: which + why

return flag_for_human_review(output, report)         # budget exhausted

Four things to watch:

• Two roles = two different calls with fully separate prompts — and ideally two different models (self-preference, above).
• The judge's report is structured output — so all_passed is a code decision, not prose parsing.
• The feedback goes into the Generator verbatim: which criterion failed + why. A "6/10" score with no details makes the next attempt a fresh guess, not a fix.
• `flag_for_human_review` is a designed path, not an exception: the output that failed 3 times is exactly the output that needs human eyes — and it arrives with a ready-made report saying why.

⚠️ Watch out — there are three stop rules, not one: an attempt cap (2–3 is usually enough), a clear pass threshold (which criteria are must-pass), and a per-task cost cap. And if you find the output hovering around the threshold round after round — 78 against a required 80, every time — don't raise the attempts: revisit the criteria themselves; most likely two items are pulling in opposite directions.

Exercise (30 minutes): Take a recurring generation task from your actual work — a product description, a support reply, a commit message. (1) Write 4 criteria, at least one checked by code (length, regex, schema). (2) Build the loop: Generator → Evaluator with a JSON report → loop capped at 3 attempts. (3) Run it on 10 inputs and record two numbers: first-attempt pass rate, and average attempts.

The first number tells you the quality of your base prompt — if 9 out of 10 pass first time, you may not need the loop at all. And if it's 3 out of 10... the Evaluator is still saving you, but the Generator needs work.

Cost considerations

> 💸 Numbers subject to change *(Volatile fragment — illustrative pricing; check your provider's current rates)*: $3/million input tokens and $15/million output tokens.

Scenario: generating a product description under the four criteria:

For 1,000 products: ~$19 — versus human review at 5 minutes per description = ~83 hours of work. And that's before counting consistency: the automated judge doesn't get tired at the end of the day and doesn't wave something through because it's in a hurry.

One number worth a spot on your dashboard: average attempts per output. If it starts creeping from 1.6 toward 2.8, something changed — the brief, the model, or the criteria — and the bill will tell you before the quality does.

• Batch generation with uniform quality: product descriptions, support replies, summaries — anything produced by the hundreds.
• A definable quality floor: you can write "must pass C1 through C4" as clear items.
• Output checkable by code: code that takes tests, data that takes schema validation — the pattern is at its strongest here.
• A pipeline running with no human in the loop but needing a real quality bar before output ships.

• A one-off task: writing the thing once? Review it with your own eyes — building an evaluator for a single output is textbook over-engineering.
• Purely subjective quality: "Is the joke funnier?" "Is the copy classier?" — if you personally can't write the criterion, the judge won't invent it for you. It will hand you scientific-looking scores for a mood-based verdict.
• The real criterion is the customer's opinion: no evaluator substitutes for the customer — get real feedback instead of simulating it.
• High-stakes decisions (legal, medical, financial): the pattern filters and triages — but the final decision must stay human. The Evaluator here is a first-pass filter, not a certification authority.

💡 Tip: A quick test before you build: write the four criteria on paper. If you wrote them in 5 minutes and all are checkable — the pattern fits. If you spent half an hour and produced things like "the text should be engaging" — your problem isn't the pattern; your problem is that you don't yet know what "good" means. And that is an important discovery in itself.

How it differs from Reflection — in a table

And the full truth: the line between them isn't a wall — it's a slope. Once you give the Reflection critic formal criteria, make the decision programmatic, and add a cap and an exit path... you have effectively built an Evaluator–Optimizer. The more explicit the criteria and the more automatic the decision, the further you've moved from "review" to "quality gate."