Evidence That an Output Is Good
Your agent has rewritten the system prompt for a feature that reads customer emails and pulls out refund requests. It pastes three emails into the chat, shows the old output and the new output for each, and the new ones are better every time. It asks to ship.
Suppose the two prompts are equally good, the three emails were fixed in advance, and each comparison is an independent coin flip with no ties. Then the new prompt wins all three with probability 0.5 cubed, which is 12.5%, so two identical prompts would produce this story about one time in eight. This agent also chose the emails after seeing the outputs, which weakens it further.
What replaces that story is an eval: a fixed set of cases, a check for each thing the output must get right, and enough repetition to tell a real difference from noise.
Any output, measured against its criteria
An eval is a set of checks that tells you whether something is good, measured against the criteria you wrote down for it. That definition covers any output: it can be a model's reply, a report an agent drafted, a store page, or a screen it built.

Each criterion gets the cheapest check that could prove it false. Three kinds of check cover most criteria, in rising order of cost:
- A code assert decides it exactly: parse, compare, count, or run the thing and see whether it works.
- A question set asks a small, fast model a few narrow yes-or-no questions, one per call.
- A judge has a stronger model score the output against a rubric written in advance, or pick the better of two outputs in a tournament.
The same ladder for three outputs that have nothing to do with prompts:
| Output | Code assert | Question set | Judge |
|---|---|---|---|
| Security report | Every finding links to a ticket | Does each finding name who owns the fix? | Rubric: is each finding's severity defensible? |
| Store page | The short description is 300 characters or fewer | Does the description name the genre? | Tournament: which of two pages sells the game better? |
| Sign-up screen | The form submits using the keyboard alone | Does each error message say how to fix the input? | Rubric on a screenshot: does it look like one design? |
The rest of this lesson works the refund extractor through, because it needs every part: a dataset, all three rungs, and outputs that change from run to run.
Why one run settles little
A unit test of a deterministic function settles a case in one run, because the same input and state give the same output. A language model samples each next token from a probability distribution, so above temperature zero the same prompt and input can produce different outputs, and even at temperature zero many hosted models do not repeat bit for bit.
So a model-backed feature has a pass rate on each input, somewhere between 0 and 1. An email the old prompt handles four times out of five will sometimes fail, and if that failure lands in a three-email demo, the new prompt looks like an improvement it never was. A judge varies the same way: score one store page twice and you may get two numbers.
Building the cases
The extractor takes an email and returns JSON with an intent (refund or other), an order ID, and an amount in cents. One case in a twenty-case dataset:
{ "id": "partial-refund-two-orders",
"email": "Hi, I ordered twice last week (A-10442 and A-10457). Please refund only the second one, the $18.50 mug arrived cracked.",
"expected": { "intent": "refund", "orderId": "A-10457", "amountCents": 1850 } }Most cases should be real production emails with personal details removed, because real traffic holds phrasings you would not think to write. Some should be past failures, each added with its correct answer when someone noticed it. The rest are hard cases you know about but have not seen fail: two orders in one email, a request in Spanish, an amount written as "eighteen fifty", an email that says "do not refund, just replace it".
Write each case's expected answer before you run the comparison that will judge a change. Outputs you explore may reveal criteria you missed; add them, then freeze them before you score held-out cases. Deciding what counted as correct after seeing the outputs grades toward whatever the model produced.
The statistics below assume the cases are independent samples of the traffic you care about; a curated challenge set measures only itself.
Cheapest check first
For the extraction, code decides everything: the output parses as JSON, the intent is one of two values, the order ID matches exactly, and the amount equals the expected integer cents, or null when the email names none.
type Expected = { intent: "refund" | "other"; orderId: string | null; amountCents: number | null };
function grade(output: string, expected: Expected): { pass: boolean; reasons: string[] } {
let parsed: unknown;
try {
parsed = JSON.parse(output);
} catch {
return { pass: false, reasons: ["output is not valid JSON"] };
}
if (typeof parsed !== "object" || parsed === null) {
return { pass: false, reasons: ["output is not a JSON object"] };
}
const got = parsed as Record<string, unknown>;
const reasons: string[] = [];
if (got.intent !== expected.intent) reasons.push(`intent ${String(got.intent)} != ${expected.intent}`);
if (got.orderId !== expected.orderId) reasons.push(`orderId ${String(got.orderId)} != ${expected.orderId}`);
if (got.amountCents !== expected.amountCents) reasons.push(`amountCents ${String(got.amountCents)} != ${expected.amountCents}`);
return { pass: reasons.length === 0, reasons };
}
const expected: Expected = { intent: "refund", orderId: "A-10457", amountCents: 1850 };
const outputs = [
'{"intent":"refund","orderId":"A-10457","amountCents":1850}',
'{"intent":"refund","orderId":"A-10442","amountCents":1850}',
'{"intent":"refund","orderId":"A-10457","amountCents":"18.50"}',
'Here is the JSON: {"intent":"refund","orderId":"A-10457","amountCents":1850}',
];
for (const output of outputs) {
const { pass, reasons } = grade(output, expected);
console.log(pass ? "pass" : `FAIL: ${reasons.join("; ")}`);
}Output
In the playground, the second output picked the wrong order, the third wrote the amount as a string of dollars, and the fourth wrapped valid JSON in a sentence, so JSON.parse throws. A judge might call each of those close enough. An assert costs nothing, never changes its verdict on the same output, and says exactly what went wrong.
Now suppose the feature also drafts a short reply. Two of its criteria are literal: it names the order and stays under 120 words. Two need reading but are narrow: does it say whether the refund was approved, and does it promise anything beyond that refund? One is taste: does the tone suit someone whose mug arrived cracked? Each rung runs only when the cheaper ones passed:
type Check = { criterion: string; pass: boolean };
// The two model calls. Each returns an answer already parsed into its type.
declare function askFast(question: string, reply: string): Promise<"yes" | "no">;
declare function judgeTone(rubric: string, email: string, reply: string): Promise<1 | 2 | 3 | 4>;
const QUESTIONS = [
{ criterion: "says whether the refund was approved", ask: "Does the reply say whether the refund was approved?", want: "yes" },
{ criterion: "promises nothing beyond the refund", ask: "Does the reply promise anything other than refunding the named order?", want: "no" },
] as const;
async function checkReply(email: string, reply: string, orderId: string, rubric: string): Promise<Check[]> {
// Rung 1: literal checks in code. A failure here ends the run before any model is asked.
const asserts: Check[] = [
{ criterion: "names the order", pass: reply.includes(orderId) },
{ criterion: "120 words or fewer", pass: reply.trim().split(/\s+/).length <= 120 },
];
if (asserts.some((c) => !c.pass)) return asserts;
// Rung 2: narrow yes-or-no questions to a small, fast model, one question per call.
const answers = await Promise.all(QUESTIONS.map((q) => askFast(q.ask, reply)));
const questions: Check[] = QUESTIONS.map((q, i) => ({ criterion: q.criterion, pass: answers[i] === q.want }));
if (questions.some((c) => !c.pass)) return [...asserts, ...questions];
// Rung 3: the one judgment of taste, scored 1 to 4 against a rubric written in advance.
const tone = await judgeTone(rubric, email, reply);
return [...asserts, ...questions, { criterion: "tone fits the complaint", pass: tone >= 3 }];
}The gate matters because an output that fails a literal check is wrong however well it reads, and a judge handed it may still score it highly. Every criterion you move from a model to code also removes an error rate you would otherwise have to measure. The question set sits in the middle because a narrow yes-or-no question is easier to answer reliably than a broad rubric, cheap enough to run on every output, and names the criterion that broke.
A judge that scores the way you do
A judge is itself a model-backed feature with its own error rate, and you measure that rate against ground truth: you, a person who understands the product, labeling outputs pass or fail.
Say you label 60 reply drafts by hand: 48 good and 12 bad. The judge passes 46 of your 48 good drafts and fails only 5 of your 12 bad ones.

Raw agreement is (46 + 5) / 60, which is 85%. A judge that says "pass" to everything agrees with you on the 48 good drafts and scores 80%. The number that matters is how many bad drafts it caught: 5 of 12, about 42%. Cohen's kappa, a standard statistic that corrects agreement for chance, is about 0.44 here: better than chance, and far below what you want before trusting the scores.
// 60 reply drafts: a person's label and the judge's verdict for each.
// The person marked 48 good and 12 bad. The judge passed 46 of the good and 7 of the bad.
type Label = "pass" | "fail";
const human: Label[] = [...Array(48).fill("pass"), ...Array(12).fill("fail")];
const judge: Label[] = [...Array(46).fill("pass"), ...Array(2).fill("fail"), ...Array(7).fill("pass"), ...Array(5).fill("fail")];
const rubberStamp: Label[] = Array(60).fill("pass");
function report(name: string, verdicts: Label[]): void {
const n = human.length;
const agree = human.filter((h, i) => h === verdicts[i]).length / n;
const bad = human.filter((h) => h === "fail").length;
const caught = human.filter((h, i) => h === "fail" && verdicts[i] === "fail").length;
const goodPassed = human.filter((h, i) => h === "pass" && verdicts[i] === "pass").length;
// Cohen's kappa: agreement beyond what two raters with these pass rates would reach by chance.
const pHuman = human.filter((h) => h === "pass").length / n;
const pJudge = verdicts.filter((v) => v === "pass").length / n;
const chance = pHuman * pJudge + (1 - pHuman) * (1 - pJudge);
const kappa = (agree - chance) / (1 - chance);
console.log(`${name}: agreement ${(agree * 100).toFixed(0)}%, good passed ${goodPassed}/${n - bad}, bad caught ${caught}/${bad}, kappa ${kappa.toFixed(2)}`);
}
report("judge ", judge);
report("rubber stamp", rubberStamp);Output
The rubber stamp scores 80% agreement with a kappa of zero, so agreement alone cannot tell you whether a judge catches anything. Check two rates separately, good outputs passed and bad outputs caught, and set the bar in advance, such as catching at least 90% of the failures you flag in a customer-facing reply.
Your labels also tune the judge toward your taste. Read the seven bad drafts it passed and write down why you failed each: it apologized three times, blamed the courier, or promised a delivery date nobody had confirmed. Put those reasons into the rubric, with a few of your labeled drafts as examples of each score. That is a judge tuned on your own ratings.

Tuning on those 60 drafts turns their agreement into a development result, because the rubric now fits them. Measure the tuned judge on drafts it never saw, from a fresh labeled set or half your labels held back from the start, and report the number of failures in that set with an uncertainty interval. Catching 11 of 12 is 92%, but an interval on 12 cases reaches well below 90%.
Judges also have known biases. The 2023 MT-Bench paper by Zheng and colleagues found that model judges tended to prefer longer answers and often favored whichever of two answers was shown first. Run every tournament pair in both orders, and check against your labels whether longer replies are winning on length alone.
Comparing two versions
Now run each prompt five times on each of the 20 cases. The old prompt passes 82 of 100 outputs and the new one 88. Before calling that an improvement, ask how big a difference two runs of the same prompt would show.
Both prompts saw the same 20 emails, so compare them case by case: the new prompt's pass rate minus the old one's. Here are the per-case pass counts out of 5:
const oldPasses = [5,5,5,5,5,5,5,5,5,5,4,4,4,4,4,3,3,3,2,1];
const newPasses = [5,5,5,5,5,5,5,5,4,5,5,5,3,5,5,4,4,2,4,2];
const RUNS = 5;
const diffs = oldPasses.map((o, i) => (newPasses[i] - o) / RUNS);
const n = diffs.length;
const mean = diffs.reduce((a, b) => a + b, 0) / n;
const variance = diffs.reduce((a, d) => a + (d - mean) ** 2, 0) / (n - 1);
const standardError = Math.sqrt(variance / n);
console.log(mean.toFixed(3), standardError.toFixed(3));
// 0.060 0.036
console.log((mean - 1.96 * standardError).toFixed(3), (mean + 1.96 * standardError).toFixed(3));
// -0.010 0.130
// The same data, ignoring that both prompts saw the same emails.
const rates = (passes: number[]) => passes.map((p) => p / RUNS);
const sampleVariance = (xs: number[]) => {
const m = xs.reduce((a, b) => a + b, 0) / xs.length;
return xs.reduce((a, x) => a + (x - m) ** 2, 0) / (xs.length - 1);
};
const unpairedSe = Math.sqrt(sampleVariance(rates(oldPasses)) / n + sampleVariance(rates(newPasses)) / n);
console.log((mean - 1.96 * unpairedSe).toFixed(3), (mean + 1.96 * unpairedSe).toFixed(3));
// -0.074 0.194
// How many cases passed on every one of the five runs.
const everyRun = (passes: number[]) => passes.filter((p) => p === RUNS).length;
console.log(everyRun(oldPasses), everyRun(newPasses));
// 10 13Output
The mean improvement is 6 points with a standard error of about 3.6, so a normal-approximation 95% interval runs from roughly −1 to +13 points, approximate with only 20 cases. Zero is inside, so this data cannot tell you whether the new prompt is better.
Pairing matters. Ignore that both prompts saw the same emails and the interval widens to about −7.4 to +19.4 points. Pairing narrows it because both prompts tend to succeed on the same emails, so the gap between easy and hard emails stops counting as noise.

Your sample size is the number of cases. Five runs of one email measure that email's pass rate more precisely and add nothing about emails you have not tested. The interval shrinks roughly with one over the square root of the number of cases, so going from 20 cases to 80 roughly halves it. An interval entirely above zero supports an improvement under the eval's assumptions; one that contains zero leaves the direction open, and shipping on it is a guess to label as one.
Does it hold up every time
A pass rate says how often one run passes. Every run of this feature reaches a customer, so also ask on how many cases all five runs passed. That measure is pass^k, here pass^5, and the playground's last line counts it: 10 of 20 cases for the old prompt and 13 of 20 for the new.
Pass^k falls fast. A case that passes 80% of the time passes five runs in a row only 0.8 to the fifth, about 33%, of the time. So 88% of outputs can pass while only 65% of cases pass on every run. Its cousin pass@k counts a case when any one of k runs passes, which suits a person picking the best of several attempts. When every output goes straight to a customer, report pass^k next to the pass rate.
An eval that can fail
If the cases are easy, every version scores near 100%, including a copy of the prompt with half its instructions deleted. Those scores show only that the eval did not distinguish the versions you tried.
Test for this by breaking the prompt on purpose: delete the instruction about amounts, or swap in a much weaker model, and rerun the eval. If the score barely moves, read the broken version's outputs on the cases that exercise what you removed. The model may still behave correctly without the instruction, or the grader may miss the failure. Confirm that the broken version actually errs, then add cases until it scores clearly worse. Keep cases the current version still fails, which can show an improvement, and cases it barely passes, which are likely to show a regression.
When every check passes and you still would not ship it
Weeks later the feature passes every check on every case, and you read a sample of replies anyway. The replies to Spanish emails came back in English. Nothing in the eval asked about language, so nothing failed.
Output that passes every eval but that you still dislike means the eval is missing a criterion. Write it down ("the reply is in the language of the email"), give it the cheapest check that works, here a question for the fast model, and rerun before you touch the prompt. The score drops, say from 20 of 20 cases to 18 of 20, and that drop is the eval getting sharper. Then fix the prompt and watch the score climb back.
Reading output by hand belongs here once the checks exist: it finds criteria nobody wrote down, and the eval then checks them on every run, so each problem needs noticing only once.
Evals as regression tests
Once the eval exists, it does for a model-backed feature what a test suite does for code. Run it on every change to the prompt, the model version, the tools the model can call, or the code that assembles its input. A model upgrade from the provider changes your system even though you did not edit a line, and it deserves the same run. Keep results per case, per version, so you can see which cases a change broke as well as whether the total moved.
When a production failure turns up, add it to the dataset with the right answer, confirm the current version fails it, and then fix the prompt. That order proves the case can detect the failure you are fixing.