Building Software
Sections

Contents Section 2, Verification and Evals

The Loop: Build, Evaluate, Iterate

Scoring a build against its written criteria, fixing what failed, scoring everything again, and sharpening the criteria themselves, one change at a time, until the build is what you wanted.

Section 2 of 9, subsection 6 of 6. Depth: do. Core 7 of 25. A 10-minute read.

Mistakes to catch in review

  1. An agent changes several things in one round, the total score rises, and nobody can say which change helped and which one broke a criterion that used to pass.

  2. Only the criteria that failed are re-run after a fix, so a regression in a criterion that used to pass goes unnoticed.

  3. A criterion the build keeps failing is loosened or deleted to reach a perfect score, with no dated reason and no evidence the original target was wrong.

  4. A criterion that needs real users is marked passed after one day of traffic, or counted as a pass while it is still pending.

  5. A build that passes every criterion but is visibly wrong in use is accepted on its score, and the missing criterion is never written.

Score It, Change One Thing, Score It Again

Your agent is building the search for a recipe site against eight written criteria. It reports back: "Round 2 done. I added a search index, stemming and fuzzy matching for typos. Score is 6 of 8, up from 4. Criterion 8 is pending until launch."

The total rises, so the round looks like progress. Put the two rounds side by side, criterion by criterion, and the picture changes. Three criteria that failed in round 1 now pass. Criterion 3, which passed in round 1, now fails. The total hides that, and with three things changed at once, nobody can say which change helped and which one broke the criterion that used to pass.

Once criteria exist, rounds like this are the unit of work: build, score, change something, score again. Run carelessly, the loop lets the score climb while the build gets worse, as round 2 just did.

What hill-climbing is

Hill-climbing is an old optimization method. Start with a candidate solution and a way to score it. Make a small change, score the result, and keep the change if the score went up. Repeat until no small change helps. Picture the score as height, and each kept change is a step uphill.

The method needs a score, and the score has to measure what you actually want. In building software, both come from the written criteria of the ideal state. Each criterion is something a stranger could check, and each carries the cheapest check that could prove it false: a line of code, a comparison against a known answer, a model judge, or a number from production. A build's score is how many criteria it passes.

Without criteria there is nothing to climb. Typing three queries and deciding the search feels better measures one person's taste on one afternoon, and the next change could undo it unnoticed.

Criteria, highlighted, labeled what you want, lead into Build, then Score, which checks every criterion cheapest first. From Score, an arrow labeled a criterion fails goes down to Change one thing, aimed at a failure, which loops back into Build. A second arrow from Score labeled all pass goes to Still wrong?, and a dashed arrow labeled yes leads to Add a criterion, watch it fail first, which points back to Criteria. Notes read the criteria set the score and one change per round.
A failing criterion sends one change back into the build. A build that passes everything and is still wrong sends a new criterion back into the criteria.

The agent can run most of the loop by itself, because each round is a change followed by a re-score. You own the criteria, which say what you want, and the judgment of whether a round's evidence means what the agent says it means.

Scoring a build

These are the eight criteria for the recipe search, each with its cheapest check.

No. Criterion Cheapest check
1 The 40 reference queries return in under 300 ms at the 95th percentile on a copy of the 12,000-recipe catalog Code, timed
2 An empty query shows the 12 recipes saved most this week Code
3 For each of the 40 reference queries, every top-3 result is one a cook would accept Reference answers
4 Each of 10 queries with one typo finds its recipe in the top 3 Reference answers
5 The singular and plural of a word return the same top 3 Code
6 A query with no matches says so and offers one suggestion that does return results Code for the suggestion, a model judge for the wording
7 The search box works from the keyboard alone and has a label a screen reader announces Automated accessibility check
8 In production, at least 45% of searches end in a click on a top-3 result, over 7 days and at least 2,000 searches Analytics, weekly

A scoring run goes in order of cost. Code checks criteria 1 to 5 and 7 in seconds, so they run on every change. The judge for criterion 6 runs only after the code part passes, since paying a model to grade the wording of a page with no working suggestion buys nothing. Criterion 8 needs real users, so before launch it is pending.

Pending is a separate result from pass, because a build that has never met a real visitor has no evidence about criterion 8 either way. In round 1, criterion 1 failed at 820 ms, criteria 4 and 5 failed, and the other four passed: 4 of 8 with 1 pending. Counting criterion 8 as a pass would claim evidence that does not exist.

Fix what failed, then score everything again

Each of the three failures has a likely cause. The search loads every recipe and filters in memory, which explains 820 ms. It matches whole words exactly, which explains the typos and the plurals.

After any fix, re-run every criterion, including the ones that already passed. A change aimed at one criterion can break another, and the passing criteria are where nobody is looking. Had the agent re-run only criteria 1, 4 and 5 after round 2, it would have reported three fixes, and the regression in criterion 3 would have gone unnoticed. The cheap checks take seconds, so skipping them saves nothing.

One change per experiment

Round 2 made three changes and moved four criteria. Any of the three changes could have broken criterion 3, or two of them together, and the data cannot tell which. So go back to round 1 and make the changes one at a time, scoring every criterion after each.

Round Change 1 2 3 4 5 6 7 Score
1 First build F P P F F P P 4/8
2 Index, stemming and fuzzy matching at once P P F P P P P 6/8
3 From round 1: index only P P P F F P P 5/8
4 Add stemming P P P F P P P 6/8
5 Add fuzzy matching P P F P P P P 6/8
Round 4, scoring 6/8, leads through an arrow labeled + fuzzy to Round 5, also 6/8. From Round 5, arrows lead to Criterion 3, highlighted, pass to fail, and Criterion 4, fail to pass. A note reads same total, two flips.
Rounds 4 and 5 both score 6/8. Adding fuzzy matching fixed criterion 4 and broke criterion 3, and only the per-criterion results show it.

Rounds 4 and 5 have the same total. A log of totals would show round 5 as a change that did nothing. It fixed criterion 4 and broke criterion 3. Fuzzy matching lets "rice" match "rich" and "ice", those recipes have more saves, and so "Rich chocolate cake" and "Lemon ice cream" push two of the three rice dishes out of the top 3.

With the cause known, the fix can aim at it. Round 6 replaces round 5's change: fuzzy matching runs only when exact matching finds fewer than three recipes, so "rice", with three exact matches, never reaches it. Round 6 scores 7 of 8, with criterion 8 still pending.

The playground is a cut-down version of the same climb: a 14-recipe catalog, the stemming and fuzzy switches (the index only affects speed), and criteria 3, 4 and 5 as code. Each round is compared criterion by criterion with the round it started from.

TypeScriptRuns in your browser
type Recipe = { title: string; saves: number };
type Build = { name: string; from?: string; stemming: boolean; fuzzy: "off" | "always" | "fallback" };

const recipes: Recipe[] = [
  { title: "Rich chocolate cake", saves: 900 }, { title: "Chicken curry", saves: 820 },
  { title: "Roast potatoes", saves: 700 }, { title: "Banana bread", saves: 650 },
  { title: "Beef lasagna", saves: 610 }, { title: "Lemon ice cream", saves: 500 },
  { title: "Tomato soup", saves: 450 }, { title: "Sourdough bread", saves: 400 },
  { title: "Potato salad", saves: 350 }, { title: "Rice pudding", saves: 300 },
  { title: "Vegetable lasagna", saves: 260 }, { title: "Broccoli cheese bake", saves: 120 },
  { title: "Spiced rice", saves: 60 }, { title: "Coconut rice", saves: 40 },
];

const stem = (w: string) =>
  w.endsWith("ies") ? w.slice(0, -3) + "y" : w.endsWith("oes") ? w.slice(0, -2) : w.endsWith("s") && !w.endsWith("ss") ? w.slice(0, -1) : w;
const words = (s: string, b: Build) =>
  s.toLowerCase().split(/[^a-z]+/).filter(Boolean).map((w) => (b.stemming ? stem(w) : w));

// Edit distance between two words: insertions, deletions and substitutions.
function distance(a: string, b: string): number {
  let prev = Array.from({ length: b.length + 1 }, (_, j) => j);
  for (let i = 1; i <= a.length; i++) {
    const row = [i];
    for (let j = 1; j <= b.length; j++)
      row[j] = Math.min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1));
    prev = row;
  }
  return prev[b.length];
}

// More matching words ranks higher; ties go to the recipe with more saves.
function rank(query: string, b: Build, fuzzy: boolean): string[] {
  const q = words(query, b);
  const near = (w: string, t: string) => w === t || (fuzzy && w.length >= 4 && distance(w, t) === 1);
  return recipes
    .map((r) => ({ r, hits: q.filter((w) => words(r.title, b).some((t) => near(w, t))).length }))
    .filter((x) => x.hits > 0)
    .sort((x, y) => y.hits - x.hits || y.r.saves - x.r.saves)
    .map((x) => x.r.title);
}

function search(query: string, b: Build): string[] {
  const exact = rank(query, b, b.fuzzy === "always");
  return b.fuzzy === "fallback" && exact.length < 3 ? rank(query, b, true) : exact;
}

// Criteria 3, 4 and 5 from the table, as code. Each returns true for pass.
const top3 = (q: string, b: Build) => search(q, b).slice(0, 3);
const criteria: Record<string, (b: Build) => boolean> = {
  "3 reference": (b) =>
    [
      { q: "rice", accept: ["Rice pudding", "Coconut rice", "Spiced rice"] },
      { q: "bread", accept: ["Banana bread", "Sourdough bread"] },
      { q: "lasagna", accept: ["Beef lasagna", "Vegetable lasagna"] },
    ].every(({ q, accept }) => top3(q, b).length > 0 && top3(q, b).every((t) => accept.includes(t))),
  "4 typos": (b) =>
    [["chiken", "Chicken curry"], ["brocoli", "Broccoli cheese bake"], ["lasagne", "Beef lasagna"]]
      .every(([q, want]) => top3(q, b).includes(want)),
  "5 plurals": (b) =>
    [["tomato", "tomatoes"], ["potato", "potatoes"]]
      .every(([one, many]) => top3(one, b).join() === top3(many, b).join()),
};

const rounds: Build[] = [
  { name: "first build", stemming: false, fuzzy: "off" },
  { name: "stemming + fuzzy at once", from: "first build", stemming: true, fuzzy: "always" },
  { name: "stemming only", from: "first build", stemming: true, fuzzy: "off" },
  { name: "+ fuzzy always", from: "stemming only", stemming: true, fuzzy: "always" },
  { name: "+ fuzzy as fallback", from: "stemming only", stemming: true, fuzzy: "fallback" },
];

// Score every criterion in every round, and compare each round with the one it started from.
const history = new Map<string, Record<string, boolean>>();
for (const b of rounds) {
  const now = Object.fromEntries(Object.entries(criteria).map(([k, check]) => [k, check(b)]));
  history.set(b.name, now);
  const base = b.from ? history.get(b.from) : undefined;
  const passed = Object.values(now).filter(Boolean).length;
  const cells = Object.keys(now).map((k) => `${k} ${now[k] ? "pass" : "FAIL"}`).join(" | ");
  const moved = base
    ? Object.keys(now).filter((k) => now[k] !== base[k]).map((k) => (now[k] ? `fixed ${k}` : `BROKE ${k}`)).join(", ")
    : "";
  console.log(`${b.name.padEnd(25)} ${passed}/3  ${cells}${moved ? `  <- ${moved}` : ""}`);
}
console.log("top 3 for rice, fuzzy always:", top3("rice", rounds[3]).join(", "));

The second line is round 2 in miniature: two criteria fixed, one broken, total up. The fourth line keeps the third line's total while two criteria trade places, and the last line shows why. Try making an exact match count for more than a fuzzy one inside rank, and see whether "fuzzy always" still breaks criterion 3.

A change here means one idea whose effect you want to know. The index is one idea and stemming is another. Two edits that only make sense together, like a new column and the code that fills it, count as one. When you cannot say which of several edits did what, split them and score each.

Slow hills

Criteria 1 to 7 move in seconds, and an afternoon is enough for a dozen experiments. Criterion 8 moves in weeks, because every measurement needs real people using the live site. A climb whose criteria need real-world data is a slow hill, and it needs different habits.

The search launches. These are the first three readings of criterion 8, with the target at 45%:

Window Searches Top-3 click rate 95% interval, roughly Result
Day 1 410 38% 4.7 points either way Pending
Week 1 4,300 41% 1.5 points either way Fail
Week 2, with a photo and cooking time in each result 4,500 49% 1.5 points either way Pass

After day 1 the agent proposed a ranking change. One day is too little data to act on, even though the simple interval already sits below 45%: searches by one person in one session are related, so the real uncertainty is wider than that interval, and a single day misses the weekly cycle of how people cook. The criterion already says so by asking for 7 days and 2,000 searches, which makes day 1 pending.

Week 1 is a fail you can trust, since its whole interval sits below 45%. Pick one change for the next week. Every result showed a title and nothing else, so a cook could not tell three lasagnas apart, and week 2 added a photo and the cooking time.

Two different weeks still differ in other ways, such as a holiday or a newsletter that brought new visitors. The stronger experiment, an A/B test, shows each layout to half the visitors in the same week. Meanwhile, keep climbing the fast criteria, and leave alone anything the slow one depends on.

When the criteria should change

Criteria are written before the build exists, by a person who keeps learning as it goes. Some of what they learn belongs in the criteria. The skill is telling a real lesson apart from pressure to make the score look good.

Stuck at round 5, the agent offered another way out: loosen criterion 3 from all 40 reference queries to 30 of 40, "since fuzzy matching adds some noise." That proposal comes entirely from what the current build does, and nothing about what cooks want has changed. Reject it and fix the build, as round 6 did. Deleting a failing criterion to report a perfect score, with no dated reason and no evidence the original target was wrong, hides the failure from everyone who reads the score.

A highlighted question, written before this build?, branches. Yes leads to Change criteria, with a date and reason. No leads to Change the build, keeping the target. A note reads then re-score everything.
Ask whether you would have written the new criterion before seeing this build. Yes changes the criteria, with a date and a reason; no changes the build.

A week into production, the search logs show that 10 of the 40 reference queries were never typed by anyone, while "air fryer" and "quick dinner" are among the top 20 real queries. Replacing those 10 with real queries is a sound change to the criteria, because it comes from evidence about what users want, even if the current build fails the new set.

One question separates the two cases: would you have written the new criterion before you saw this build? If yes, change the criterion. If it exists only because the build cannot meet the old one, change the build. Either way, record the change in the ideal state's decisions log with the date and the reason, and re-score the current build against the new criteria.

Passing everything and still wrong

At 8 of 8, a tester signs in with a second account and searches "curry". The third result is "Grandma's curry", a recipe another user marked private. Every criterion passes, and the search is still wrong.

Daniel Miessler names this signal in "General Hill-Climbing": "The best way to tell if you need to update your evals is if you get a result that perfectly passes your evals, but you still don't like it for some reason." A result like this means a criterion is missing. The eight criteria describe good results and say nothing about whose recipes may appear in them.

A sequence: 8/8, every criterion passes, leads to Private recipe, found by a tester, then to Criterion 9, no private results, then to 8/9, highlighted, the new check fails. From there an arrow leads down to Filter by visibility, the fix, and then to 9/9, checked on every change. A note reads the drop is progress.
Adding the missing criterion drops the score from 8/8 to 8/9 before any code changes. That drop proves the new check sees the leak, and the fix brings the score to 9/9.

Write the missing criterion so a stranger could check it: for each of 20 test accounts, no search returns a recipe its owner marked private. Run it against the current build before anyone touches the code. It fails, and the score drops to 8 of 9. That drop is progress: the criteria are sharper, and the failing run proves the new check can see the problem. The agent filters results by visibility in the query, and the score returns to 9 of 9.

A fix with no criterion leaves the next change free to leak private recipes under a perfect score. Accepting the build on its score, when you can see it is wrong in use, is the same mistake one step earlier.

Keeping the score history

The record of a climb is a log with one line per round: what changed, the result for each criterion, and the score. Changes to the criteria get their own lines, with the reason in the decisions log.

Output
round  change                                        1 2 3 4 5 6 7 8 9  score
1      first build                                   F P P F F P P -    4/8
2      index + stemming + fuzzy (agent, at once)     P P F P P P P -    6/8
3      from 1: index only                            P P P F F P P -    5/8
4      + stemming                                    P P P F P P P -    6/8
5      + fuzzy matching                              P P F P P P P -    6/8
6      fuzzy as fallback only (replaces 5)           P P P P P P P -    7/8
7      launch, week 1                                P P P P P P P F    7/8
8      photo and cooking time in each result         P P P P P P P P    8/8
9      criterion 3 revised: 10 queries from logs     P P P P P P P P    8/8
10     criterion 9 added: no private recipes         P P P P P P P P F  8/9
11     filter results by visibility                  P P P P P P P P P  9/9

A row shows what one change did. A column shows when a criterion first passed and whether it ever broke again. The totals alone hide the regression in round 5, and they make round 10 look like a step backward when it was the criteria getting sharper.

A climb you can trust changes one thing per round, re-scores every criterion and writes the result down. Its criteria change only for a reason you would have accepted before seeing the build, and a perfect score on a build you can see is wrong means the next thing to write is a criterion.

Check your understanding

7 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An agent is climbing an invoice PDF generator against five criteria. It reports round 7 as "no change in score, reverting the streaming renderer." What does the log show?
    Output
    round  change                        1 2 3 4 5  score
    6      baseline                      P F P P F  3/5
    7      switch to streaming renderer  P P P F F  3/5
    Show the answer

    Answer C. The columns show what the total hides: criterion 2 went from F to P and criterion 4 from P to F. Reverting throws away a real fix along with the regression. The next step is to find which part of the renderer change broke criterion 4 and aim a narrower change at it, then score every criterion again.

  2. 2An agent wrote this scoring function for a build whose fourth criterion needs a week of production traffic. What does it print, and is the score honest?
    TypeScript
    type Result = "pass" | "fail" | "pending";
    
    function score(results: Result[]): string {
      const passed = results.filter((r) => r !== "fail").length;
      return `${passed}/${results.length}`;
    }
    
    console.log(score(["pass", "fail", "pass", "pending", "pass"]));
    Show the answer

    Answer A. The filter keeps everything that is not a fail, so the pending result counts as a pass and the function prints 4/5. A pending criterion has no evidence either way. Report it separately, as 3 of 5 with 1 pending, and count only real passes.

  3. 3After each fix, an agent's loop re-scores the build with this function. What can it miss?
    TypeScript
    // Re-check whatever failed last round.
    function rescore(last: Record<string, boolean>, checks: Record<string, () => boolean>) {
      for (const k of Object.keys(last)) {
        if (!last[k]) last[k] = checks[k]();
      }
      return last;
    }
    Show the answer

    Answer B. Only criteria that failed last round are checked again, so a passing criterion keeps its old result even if the fix broke it. A fix aimed at one criterion can break another. Cheap checks take seconds, so re-run every criterion after every change.

  4. 4A criterion says at least 60% of started signups complete, measured over 14 days and at least 3,000 starts. Two days after launch, 57% of 700 starts have completed, and the agent wants to redesign the form now. What do you do?
    Show the answer

    Answer D. At 700 starts, a 95% interval around 57% runs roughly 53% to 61%, so the data cannot yet say the target is missed, and two days miss the weekly cycle the 14 days were chosen to cover. Changing the form mid-reading would also mix two versions into one number. Wait for the reading the criterion names.

  5. 5Which of these is a sound reason to change a criterion instead of the build?
    Show the answer

    Answer B. Evidence about what users actually do is something you learned about what you want, so you would have written the new criterion before seeing this build. Repeated failure only says the build cannot meet the target yet. Change the build, or record a dated reason when the target itself was wrong.

  6. 6A password reset flow passes all 12 of its criteria. Testing it yourself, you notice the reset email arrives in Spanish for an account set to English. What comes next?
    Show the answer

    Answer C. A build that passes every criterion and is still wrong points at a missing criterion. Writing it and watching today's build fail it proves the check can see the problem, and from then on every change is checked for it. A bare fix leaves the next change free to bring the bug back under a perfect score.

  7. 7An agent wants to test several edits in one round. Which pair can reasonably be scored as a single change?
    Show the answer

    Answer A. The setting does nothing until code reads it, so the two edits are one idea with one effect to measure. The other pairs are separate ideas that can each help or hurt on their own. Scored together, a gain from one can hide a loss from the other, and the log cannot say which did what.

You understand it when you can

Drill

An agent's climb log for a CSV order import reads 6/9 in round 1 and 9/9 in round 2. In round 2 it switched parsing libraries, added a retry loop around each database write, raised the request timeout, and deleted criterion 7 ('a row with a negative quantity is rejected and reported') with the note 'flaky'. The day after release, a customer's import creates every order twice. Say what the log cannot tell you, which changes you would rerun alone and in what order, what should happen to criterion 7, and write the criterion the climb was missing.

Topics

Scoring a Build Against Its Criteria
Running every criterion's check, cheapest first, and recording pass, fail or pending for each one, so a score claims only the evidence it has.
Fix What Failed, Then Re-score Everything
Repairing the failed criteria and re-running the whole set, because a fix aimed at one criterion can break another that used to pass.
One Change per Experiment
Changing one idea between scorings and comparing every criterion with the round it started from, so each move up or down the hill has a known cause.
Slow Hills
Criteria that need real users or real traffic: waiting for the duration and volume the criterion names, comparing fairly, and climbing the fast criteria meanwhile.
Changing the Criteria Instead of the Build
Changing a target only for something learned about what you want, never because the build cannot meet it, and adding the missing criterion when a build passes everything but is still wrong.
Score History
A log with one line per round: the change, each criterion's result and the score, plus every change to the criteria with its date and reason.

Further reading

Start here

Short reads

Watch

Books

  • Experimentation Works: The Surprising Power of Business Experiments. Stefan H. Thomke, 2020.

    Makes the case, with cases from large online companies, for testing changes one controlled experiment at a time instead of trusting judgment about which idea will win.

  • How to Measure Anything: Finding the Value of Intangibles in Business. Douglas W. Hubbard, 2014, 3rd edition.

    Shows how to turn something that seems unmeasurable into an observable quantity with a stated uncertainty, which is the work of writing a criterion a stranger could check.

Primary sources