Building Software
Sections

Contents Section 2, Verification and Evals

Evals: Measuring Any Output Against Its Criteria

Measuring any output an agent produces, from a model's reply to a report or a page, against its written criteria with the cheapest check that works, so a change is judged on evidence.

Section 2 of 9, subsection 5 of 6. Depth: do. Core 6 of 25. A 10-minute read.

Mistakes to catch in review

  1. A change shipped because its output looked better on three hand-picked examples.

  2. A model judge asked to decide something code could check exactly, such as a format, a field or a length.

  3. A judge's scores trusted without measuring how many of your own failing labels it catches.

  4. An eval set so easy that every version scores near perfect, so it cannot detect a regression.

  5. A feature called reliable from its pass rate, when far fewer cases pass on every run.

  6. Output that passes every eval but that you still would not ship, accepted instead of turned into a new criterion.

Evidence That an Output Is Good

Your agent has rewritten the system prompt for a feature that reads customer emails and pulls out refund requests. It pastes three emails into the chat, shows the old output and the new output for each, and the new ones are better every time. It asks to ship.

Suppose the two prompts are equally good, the three emails were fixed in advance, and each comparison is an independent coin flip with no ties. Then the new prompt wins all three with probability 0.5 cubed, which is 12.5%, so two identical prompts would produce this story about one time in eight. This agent also chose the emails after seeing the outputs, which weakens it further.

What replaces that story is an eval: a fixed set of cases, a check for each thing the output must get right, and enough repetition to tell a real difference from noise.

Any output, measured against its criteria

An eval is a set of checks that tells you whether something is good, measured against the criteria you wrote down for it. That definition covers any output: it can be a model's reply, a report an agent drafted, a store page, or a screen it built.

An output of any kind goes first to a highlighted code assert, which costs nothing, then to a question set asked of a small fast model, then to a judge, a stronger model, and finally to pass. Each move to the next rung happens only after a pass. Below each of the three rungs, a fail mark says the output stops there. A note reads literal checks gate judged ones.
An output climbs the rungs in order of cost, and a failure at any rung stops it there, so no model grades an output that already failed a literal check.

Each criterion gets the cheapest check that could prove it false. Three kinds of check cover most criteria, in rising order of cost:

  • A code assert decides it exactly: parse, compare, count, or run the thing and see whether it works.
  • A question set asks a small, fast model a few narrow yes-or-no questions, one per call.
  • A judge has a stronger model score the output against a rubric written in advance, or pick the better of two outputs in a tournament.

The same ladder for three outputs that have nothing to do with prompts:

Output Code assert Question set Judge
Security report Every finding links to a ticket Does each finding name who owns the fix? Rubric: is each finding's severity defensible?
Store page The short description is 300 characters or fewer Does the description name the genre? Tournament: which of two pages sells the game better?
Sign-up screen The form submits using the keyboard alone Does each error message say how to fix the input? Rubric on a screenshot: does it look like one design?

The rest of this lesson works the refund extractor through, because it needs every part: a dataset, all three rungs, and outputs that change from run to run.

Why one run settles little

A unit test of a deterministic function settles a case in one run, because the same input and state give the same output. A language model samples each next token from a probability distribution, so above temperature zero the same prompt and input can produce different outputs, and even at temperature zero many hosted models do not repeat bit for bit.

So a model-backed feature has a pass rate on each input, somewhere between 0 and 1. An email the old prompt handles four times out of five will sometimes fail, and if that failure lands in a three-email demo, the new prompt looks like an improvement it never was. A judge varies the same way: score one store page twice and you may get two numbers.

Building the cases

The extractor takes an email and returns JSON with an intent (refund or other), an order ID, and an amount in cents. One case in a twenty-case dataset:

JSON
{ "id": "partial-refund-two-orders",
  "email": "Hi, I ordered twice last week (A-10442 and A-10457). Please refund only the second one, the $18.50 mug arrived cracked.",
  "expected": { "intent": "refund", "orderId": "A-10457", "amountCents": 1850 } }

Most cases should be real production emails with personal details removed, because real traffic holds phrasings you would not think to write. Some should be past failures, each added with its correct answer when someone noticed it. The rest are hard cases you know about but have not seen fail: two orders in one email, a request in Spanish, an amount written as "eighteen fifty", an email that says "do not refund, just replace it".

Write each case's expected answer before you run the comparison that will judge a change. Outputs you explore may reveal criteria you missed; add them, then freeze them before you score held-out cases. Deciding what counted as correct after seeing the outputs grades toward whatever the model produced.

The statistics below assume the cases are independent samples of the traffic you care about; a curated challenge set measures only itself.

Cheapest check first

For the extraction, code decides everything: the output parses as JSON, the intent is one of two values, the order ID matches exactly, and the amount equals the expected integer cents, or null when the email names none.

TypeScriptRuns in your browser
type Expected = { intent: "refund" | "other"; orderId: string | null; amountCents: number | null };

function grade(output: string, expected: Expected): { pass: boolean; reasons: string[] } {
  let parsed: unknown;
  try {
    parsed = JSON.parse(output);
  } catch {
    return { pass: false, reasons: ["output is not valid JSON"] };
  }
  if (typeof parsed !== "object" || parsed === null) {
    return { pass: false, reasons: ["output is not a JSON object"] };
  }
  const got = parsed as Record<string, unknown>;
  const reasons: string[] = [];
  if (got.intent !== expected.intent) reasons.push(`intent ${String(got.intent)} != ${expected.intent}`);
  if (got.orderId !== expected.orderId) reasons.push(`orderId ${String(got.orderId)} != ${expected.orderId}`);
  if (got.amountCents !== expected.amountCents) reasons.push(`amountCents ${String(got.amountCents)} != ${expected.amountCents}`);
  return { pass: reasons.length === 0, reasons };
}

const expected: Expected = { intent: "refund", orderId: "A-10457", amountCents: 1850 };
const outputs = [
  '{"intent":"refund","orderId":"A-10457","amountCents":1850}',
  '{"intent":"refund","orderId":"A-10442","amountCents":1850}',
  '{"intent":"refund","orderId":"A-10457","amountCents":"18.50"}',
  'Here is the JSON: {"intent":"refund","orderId":"A-10457","amountCents":1850}',
];
for (const output of outputs) {
  const { pass, reasons } = grade(output, expected);
  console.log(pass ? "pass" : `FAIL: ${reasons.join("; ")}`);
}

In the playground, the second output picked the wrong order, the third wrote the amount as a string of dollars, and the fourth wrapped valid JSON in a sentence, so JSON.parse throws. A judge might call each of those close enough. An assert costs nothing, never changes its verdict on the same output, and says exactly what went wrong.

Now suppose the feature also drafts a short reply. Two of its criteria are literal: it names the order and stays under 120 words. Two need reading but are narrow: does it say whether the refund was approved, and does it promise anything beyond that refund? One is taste: does the tone suit someone whose mug arrived cracked? Each rung runs only when the cheaper ones passed:

TypeScript
type Check = { criterion: string; pass: boolean };

// The two model calls. Each returns an answer already parsed into its type.
declare function askFast(question: string, reply: string): Promise<"yes" | "no">;
declare function judgeTone(rubric: string, email: string, reply: string): Promise<1 | 2 | 3 | 4>;

const QUESTIONS = [
  { criterion: "says whether the refund was approved", ask: "Does the reply say whether the refund was approved?", want: "yes" },
  { criterion: "promises nothing beyond the refund", ask: "Does the reply promise anything other than refunding the named order?", want: "no" },
] as const;

async function checkReply(email: string, reply: string, orderId: string, rubric: string): Promise<Check[]> {
  // Rung 1: literal checks in code. A failure here ends the run before any model is asked.
  const asserts: Check[] = [
    { criterion: "names the order", pass: reply.includes(orderId) },
    { criterion: "120 words or fewer", pass: reply.trim().split(/\s+/).length <= 120 },
  ];
  if (asserts.some((c) => !c.pass)) return asserts;

  // Rung 2: narrow yes-or-no questions to a small, fast model, one question per call.
  const answers = await Promise.all(QUESTIONS.map((q) => askFast(q.ask, reply)));
  const questions: Check[] = QUESTIONS.map((q, i) => ({ criterion: q.criterion, pass: answers[i] === q.want }));
  if (questions.some((c) => !c.pass)) return [...asserts, ...questions];

  // Rung 3: the one judgment of taste, scored 1 to 4 against a rubric written in advance.
  const tone = await judgeTone(rubric, email, reply);
  return [...asserts, ...questions, { criterion: "tone fits the complaint", pass: tone >= 3 }];
}

The gate matters because an output that fails a literal check is wrong however well it reads, and a judge handed it may still score it highly. Every criterion you move from a model to code also removes an error rate you would otherwise have to measure. The question set sits in the middle because a narrow yes-or-no question is easier to answer reliably than a broad rubric, cheap enough to run on every output, and names the criterion that broke.

A judge that scores the way you do

A judge is itself a model-backed feature with its own error rate, and you measure that rate against ground truth: you, a person who understands the product, labeling outputs pass or fail.

Say you label 60 reply drafts by hand: 48 good and 12 bad. The judge passes 46 of your 48 good drafts and fails only 5 of your 12 bad ones.

60 drafts labeled by hand split into 48 good and 12 bad. Of the 48 good, the judge passes 46 and fails 2. Of the 12 bad, it passes 7, highlighted as failures let through, and catches 5, about 42%. Dashed lines join the 46 passed and the 5 caught to 85% agree, (46 + 5) / 60. A note reads it lets most failures through.
Most of the agreement comes from the 46 good drafts the judge passes. Of the 12 drafts a person marked bad, the judge passes 7 and catches only 5.

Raw agreement is (46 + 5) / 60, which is 85%. A judge that says "pass" to everything agrees with you on the 48 good drafts and scores 80%. The number that matters is how many bad drafts it caught: 5 of 12, about 42%. Cohen's kappa, a standard statistic that corrects agreement for chance, is about 0.44 here: better than chance, and far below what you want before trusting the scores.

TypeScriptRuns in your browser
// 60 reply drafts: a person's label and the judge's verdict for each.
// The person marked 48 good and 12 bad. The judge passed 46 of the good and 7 of the bad.
type Label = "pass" | "fail";
const human: Label[] = [...Array(48).fill("pass"), ...Array(12).fill("fail")];
const judge: Label[] = [...Array(46).fill("pass"), ...Array(2).fill("fail"), ...Array(7).fill("pass"), ...Array(5).fill("fail")];
const rubberStamp: Label[] = Array(60).fill("pass");

function report(name: string, verdicts: Label[]): void {
  const n = human.length;
  const agree = human.filter((h, i) => h === verdicts[i]).length / n;
  const bad = human.filter((h) => h === "fail").length;
  const caught = human.filter((h, i) => h === "fail" && verdicts[i] === "fail").length;
  const goodPassed = human.filter((h, i) => h === "pass" && verdicts[i] === "pass").length;
  // Cohen's kappa: agreement beyond what two raters with these pass rates would reach by chance.
  const pHuman = human.filter((h) => h === "pass").length / n;
  const pJudge = verdicts.filter((v) => v === "pass").length / n;
  const chance = pHuman * pJudge + (1 - pHuman) * (1 - pJudge);
  const kappa = (agree - chance) / (1 - chance);
  console.log(`${name}: agreement ${(agree * 100).toFixed(0)}%, good passed ${goodPassed}/${n - bad}, bad caught ${caught}/${bad}, kappa ${kappa.toFixed(2)}`);
}

report("judge       ", judge);
report("rubber stamp", rubberStamp);

The rubber stamp scores 80% agreement with a kappa of zero, so agreement alone cannot tell you whether a judge catches anything. Check two rates separately, good outputs passed and bad outputs caught, and set the bar in advance, such as catching at least 90% of the failures you flag in a customer-facing reply.

Your labels also tune the judge toward your taste. Read the seven bad drafts it passed and write down why you failed each: it apologized three times, blamed the courier, or promised a delivery date nobody had confirmed. Put those reasons into the rubric, with a few of your labeled drafts as examples of each score. That is a judge tuned on your own ratings.

Your labels, 60 drafts, feed tune rubric, with your reasons and examples, which produces the highlighted tuned judge. A dashed line goes from the tuned judge to same 60, a development result. A solid line goes to fresh drafts, never seen, where it catches 11 of 12. A note reads measure on unseen drafts.
Your labels tune the rubric, so the tuned judge's score on those same 60 drafts is a development result. Drafts it never saw measure it: there it catches 11 of 12, and with 12 failures the interval still reaches well below 90%.

Tuning on those 60 drafts turns their agreement into a development result, because the rubric now fits them. Measure the tuned judge on drafts it never saw, from a fresh labeled set or half your labels held back from the start, and report the number of failures in that set with an uncertainty interval. Catching 11 of 12 is 92%, but an interval on 12 cases reaches well below 90%.

Judges also have known biases. The 2023 MT-Bench paper by Zheng and colleagues found that model judges tended to prefer longer answers and often favored whichever of two answers was shown first. Run every tournament pair in both orders, and check against your labels whether longer replies are winning on length alone.

Comparing two versions

Now run each prompt five times on each of the 20 cases. The old prompt passes 82 of 100 outputs and the new one 88. Before calling that an improvement, ask how big a difference two runs of the same prompt would show.

Both prompts saw the same 20 emails, so compare them case by case: the new prompt's pass rate minus the old one's. Here are the per-case pass counts out of 5:

TypeScriptRuns in your browser
const oldPasses = [5,5,5,5,5,5,5,5,5,5,4,4,4,4,4,3,3,3,2,1];
const newPasses = [5,5,5,5,5,5,5,5,4,5,5,5,3,5,5,4,4,2,4,2];
const RUNS = 5;

const diffs = oldPasses.map((o, i) => (newPasses[i] - o) / RUNS);
const n = diffs.length;
const mean = diffs.reduce((a, b) => a + b, 0) / n;
const variance = diffs.reduce((a, d) => a + (d - mean) ** 2, 0) / (n - 1);
const standardError = Math.sqrt(variance / n);

console.log(mean.toFixed(3), standardError.toFixed(3));
// 0.060 0.036
console.log((mean - 1.96 * standardError).toFixed(3), (mean + 1.96 * standardError).toFixed(3));
// -0.010 0.130

// The same data, ignoring that both prompts saw the same emails.
const rates = (passes: number[]) => passes.map((p) => p / RUNS);
const sampleVariance = (xs: number[]) => {
  const m = xs.reduce((a, b) => a + b, 0) / xs.length;
  return xs.reduce((a, x) => a + (x - m) ** 2, 0) / (xs.length - 1);
};
const unpairedSe = Math.sqrt(sampleVariance(rates(oldPasses)) / n + sampleVariance(rates(newPasses)) / n);
console.log((mean - 1.96 * unpairedSe).toFixed(3), (mean + 1.96 * unpairedSe).toFixed(3));
// -0.074 0.194

// How many cases passed on every one of the five runs.
const everyRun = (passes: number[]) => passes.filter((p) => p === RUNS).length;
console.log(everyRun(oldPasses), everyRun(newPasses));
// 10 13

The mean improvement is 6 points with a standard error of about 3.6, so a normal-approximation 95% interval runs from roughly −1 to +13 points, approximate with only 20 cases. Zero is inside, so this data cannot tell you whether the new prompt is better.

Pairing matters. Ignore that both prompts saw the same emails and the interval widens to about −7.4 to +19.4 points. Pairing narrows it because both prompts tend to succeed on the same emails, so the gap between easy and hard emails stops counting as noise.

A number line of new-minus-old pass rate in points, from −10 to +20. Above it, a purple bar labelled 'Paired' spans −1 to +13, and a wider bar labelled 'Unpaired' spans −7.4 to +19.4. A dashed line drops from 'Zero' through both bars to 0 on the axis. Notes read 'zero is inside both' and 'pairing narrows the interval'.
Both intervals for the new prompt's improvement contain zero, so the data cannot say whether it is better. Pairing by email narrows the interval from −7.4 to +19.4 points down to −1 to +13.

Your sample size is the number of cases. Five runs of one email measure that email's pass rate more precisely and add nothing about emails you have not tested. The interval shrinks roughly with one over the square root of the number of cases, so going from 20 cases to 80 roughly halves it. An interval entirely above zero supports an improvement under the eval's assumptions; one that contains zero leaves the direction open, and shipping on it is a guess to label as one.

Does it hold up every time

A pass rate says how often one run passes. Every run of this feature reaches a customer, so also ask on how many cases all five runs passed. That measure is pass^k, here pass^5, and the playground's last line counts it: 10 of 20 cases for the old prompt and 13 of 20 for the new.

Pass^k falls fast. A case that passes 80% of the time passes five runs in a row only 0.8 to the fifth, about 33%, of the time. So 88% of outputs can pass while only 65% of cases pass on every run. Its cousin pass@k counts a case when any one of k runs passes, which suits a person picking the best of several attempts. When every output goes straight to a customer, report pass^k next to the pass rate.

An eval that can fail

If the cases are easy, every version scores near 100%, including a copy of the prompt with half its instructions deleted. Those scores show only that the eval did not distinguish the versions you tried.

Test for this by breaking the prompt on purpose: delete the instruction about amounts, or swap in a much weaker model, and rerun the eval. If the score barely moves, read the broken version's outputs on the cases that exercise what you removed. The model may still behave correctly without the instruction, or the grader may miss the failure. Confirm that the broken version actually errs, then add cases until it scores clearly worse. Keep cases the current version still fails, which can show an improvement, and cases it barely passes, which are likely to show a regression.

When every check passes and you still would not ship it

Weeks later the feature passes every check on every case, and you read a sample of replies anyway. The replies to Spanish emails came back in English. Nothing in the eval asked about language, so nothing failed.

Output that passes every eval but that you still dislike means the eval is missing a criterion. Write it down ("the reply is in the language of the email"), give it the cheapest check that works, here a question for the fast model, and rerun before you touch the prompt. The score drops, say from 20 of 20 cases to 18 of 20, and that drop is the eval getting sharper. Then fix the prompt and watch the score climb back.

Reading output by hand belongs here once the checks exist: it finds criteria nobody wrote down, and the eval then checks them on every run, so each problem needs noticing only once.

Evals as regression tests

Once the eval exists, it does for a model-backed feature what a test suite does for code. Run it on every change to the prompt, the model version, the tools the model can call, or the code that assembles its input. A model upgrade from the provider changes your system even though you did not edit a line, and it deserves the same run. Keep results per case, per version, so you can see which cases a change broke as well as whether the total moved.

When a production failure turns up, add it to the dataset with the right answer, confirm the current version fails it, and then fix the prompt. That order proves the case can detect the failure you are fixing.

Check your understanding

8 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1Your eval for an agent-written onboarding email has three criteria: the link to /start returns 200, the email names the customer's plan, and the tone is warm. The agent sends every email to a strong judge with all three criteria in one rubric. What should change?
    Show the answer

    Answer B. A link that returns 200 and a plan name that appears in the text each have one right answer, so code settles them for free and the same way every time. Only tone needs a model, and it should run only on emails that passed the literal checks, because a judge can call a broken email warm. Averaging judge runs reduces noise but still pays a model to settle what a request and a string match already settle.

  2. 2An agent wrote this grader for the refund extractor. The case's email mentions orders A-10442 and A-10457, and the expected order is A-10457. What is wrong with the grader?
    TypeScript
    function grade(output: string, expected: { orderId: string }): boolean {
      return output.includes(expected.orderId);
    }
    
    // One model output for this case:
    // {"intent":"refund","orderId":"A-10442","note":"also mentions A-10457"}
    Show the answer

    Answer A. The grader checks that the string appears somewhere, so the output above passes while naming the wrong order. Parse the JSON and compare the orderId field exactly. The grader is deterministic: the model's output varies between runs, but the same output always gets the same verdict.

  3. 3A person labeled 100 judge outputs: 90 good, 10 bad. The judge passed 89 of the good ones and failed 2 of the bad ones, and the agent reports 91% agreement. What is the right reading?
    Show the answer

    Answer D. A judge that passes every output would agree with the person on all 90 good ones and score 90%. This one catches 2 of the 10 failures, so 8 of every 10 bad outputs get through. Agreement looks high because most outputs are good; the catch rate on bad outputs is the number to check.

  4. 4You labeled 80 reply drafts, found the judge missed most of the bad ones, and rewrote its rubric using your reasons plus a few of those drafts as examples. On the same 80 drafts it now catches 19 of 20 bad ones. What does that tell you?
    Show the answer

    Answer C. The rubric was written to fit these 80 drafts, and some of them sit inside it as examples, so its score on them is a development result. Only labeled drafts it never saw show how it grades new ones, and even there 19 of 20 carries an interval that reaches well below 90%, so report the count beside the rate.

  5. 5Two prompts ran on the same 30 cases, five runs each. This code computes the paired difference in pass rate (new minus old) and prints the result shown. What do you conclude?
    TypeScript
    console.log(mean.toFixed(3), standardError.toFixed(3));
    // 0.040 0.025
    Show the answer

    Answer B. The mean plus or minus 1.96 standard errors is about -0.009 to +0.089, and zero is inside. More runs per case pin down each case's pass rate more precisely but add nothing about cases you have not tested, so the usual way to narrow the interval is more independent cases.

  6. 6An agent reports that its new prompt passes 90% of runs, from five runs on each of ten cases. Every output goes straight to a customer. From the log below, on what share of cases did every run pass?
    TypeScript
    // passes out of 5 runs, one entry per case
    const passes = [5, 5, 4, 5, 4, 5, 4, 5, 4, 4];
    // total: 45 of 50 runs passed
    Show the answer

    Answer D. Pass^5 counts the cases where all five runs passed, and that is the five cases logged as 5. Raising 0.9 to the fifth assumes every case fails at the same rate, but here the failures sit on five cases that each fail one run in five. 100% is pass@5, which fits only when a person picks the best of several attempts.

  7. 7Your 40-case eval scores 100% on the current prompt, 100% on the new model, and 100% on a copy of the prompt with the instructions about amounts deleted. What do you do next?
    Show the answer

    Answer C. A perfect score on a deliberately broken prompt means the eval did not tell the versions apart. Read the broken version's outputs on amount cases first: if it still gets them right, the cases do not need the instruction; if it errs and still passes, the grader is missing it. More runs may expose a rare intermittent failure, but they do not tell you whether the broken copy really errs or whether the grader would see it.

  8. 8An agent drafts incident reports, and every draft now passes all 12 of your eval's checks. Reading three of them, you notice none says when customers were first affected. What do you do first?
    Show the answer

    Answer A. Output that passes every check but still falls short means a criterion is missing. Writing it down with its cheapest check makes the eval fail these drafts, which proves the check can see the problem. Changing the instructions first and rerunning the old 12 checks would pass whether or not the fix worked, because none of them asks about impact start.

You understand it when you can

Drill

An agent rewrote the prompt behind your app's weekly account-summary emails and reports that a model judge, using a rubric the agent wrote, scores the new emails 4.6 out of 5 against 4.2 for the old, on ten accounts it picked, one run each. Find every reason this does not show the new emails are better, and write the eval that could: its criteria with the cheapest check for each, how you would know the judge scores the way you do, and how many cases and runs it needs.

Topics

Evals for Any Output
Measuring a model's reply, a report, a page or a screen against the criteria written down for it.
Eval Datasets
Building test cases with expected outcomes from real usage and past failures.
The Check Ladder: Assert, Question Set, Judge
Giving each criterion the cheapest check that could prove it false, with code asserts gating every check a model makes.
Calibrating Judges on Your Own Ratings
Measuring a judge against your labels, tuning it on them, and re-measuring it on labels it never saw.
Variance, Pass Rates and Pass^k
Running each case several times, comparing versions case by case, and counting the cases that pass on every run.
When Every Check Passes and It Still Falls Short
Treating output you would not ship despite a perfect score as a missing criterion, and adding its check.
Evals as Regression Tests
Running evals on every prompt, model or tool change, the way tests run on every code change.

Further reading

Start here

  • How to Automate AI Evals (Correctly). Hamel Husain, 2026. 27-minute explainer.

    Argues for reading and labelling real traces before writing any rubric, so the eval set is built from the failures that actually happen instead of cases the author picked.

Short reads

Watch

Books

  • AI Engineering: Building Applications with Foundation Models. Chip Huyen, 2025.

    Two chapters on evaluation cover exact-match and similarity metrics, AI-as-a-judge and its failure modes, and how to design an evaluation pipeline for an open-ended model feature.

  • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Ron Kohavi, Diane Tang and Ya Xu, 2020.

    The standard text on variance, statistical power and false positives when comparing two versions, which is exactly the discipline a prompt A/B comparison needs.

Primary sources