Score It, Change One Thing, Score It Again
Your agent is building the search for a recipe site against eight written criteria. It reports back: "Round 2 done. I added a search index, stemming and fuzzy matching for typos. Score is 6 of 8, up from 4. Criterion 8 is pending until launch."
The total rises, so the round looks like progress. Put the two rounds side by side, criterion by criterion, and the picture changes. Three criteria that failed in round 1 now pass. Criterion 3, which passed in round 1, now fails. The total hides that, and with three things changed at once, nobody can say which change helped and which one broke the criterion that used to pass.
Once criteria exist, rounds like this are the unit of work: build, score, change something, score again. Run carelessly, the loop lets the score climb while the build gets worse, as round 2 just did.
What hill-climbing is
Hill-climbing is an old optimization method. Start with a candidate solution and a way to score it. Make a small change, score the result, and keep the change if the score went up. Repeat until no small change helps. Picture the score as height, and each kept change is a step uphill.
The method needs a score, and the score has to measure what you actually want. In building software, both come from the written criteria of the ideal state. Each criterion is something a stranger could check, and each carries the cheapest check that could prove it false: a line of code, a comparison against a known answer, a model judge, or a number from production. A build's score is how many criteria it passes.
Without criteria there is nothing to climb. Typing three queries and deciding the search feels better measures one person's taste on one afternoon, and the next change could undo it unnoticed.

The agent can run most of the loop by itself, because each round is a change followed by a re-score. You own the criteria, which say what you want, and the judgment of whether a round's evidence means what the agent says it means.
Scoring a build
These are the eight criteria for the recipe search, each with its cheapest check.
| No. | Criterion | Cheapest check |
|---|---|---|
| 1 | The 40 reference queries return in under 300 ms at the 95th percentile on a copy of the 12,000-recipe catalog | Code, timed |
| 2 | An empty query shows the 12 recipes saved most this week | Code |
| 3 | For each of the 40 reference queries, every top-3 result is one a cook would accept | Reference answers |
| 4 | Each of 10 queries with one typo finds its recipe in the top 3 | Reference answers |
| 5 | The singular and plural of a word return the same top 3 | Code |
| 6 | A query with no matches says so and offers one suggestion that does return results | Code for the suggestion, a model judge for the wording |
| 7 | The search box works from the keyboard alone and has a label a screen reader announces | Automated accessibility check |
| 8 | In production, at least 45% of searches end in a click on a top-3 result, over 7 days and at least 2,000 searches | Analytics, weekly |
A scoring run goes in order of cost. Code checks criteria 1 to 5 and 7 in seconds, so they run on every change. The judge for criterion 6 runs only after the code part passes, since paying a model to grade the wording of a page with no working suggestion buys nothing. Criterion 8 needs real users, so before launch it is pending.
Pending is a separate result from pass, because a build that has never met a real visitor has no evidence about criterion 8 either way. In round 1, criterion 1 failed at 820 ms, criteria 4 and 5 failed, and the other four passed: 4 of 8 with 1 pending. Counting criterion 8 as a pass would claim evidence that does not exist.
Fix what failed, then score everything again
Each of the three failures has a likely cause. The search loads every recipe and filters in memory, which explains 820 ms. It matches whole words exactly, which explains the typos and the plurals.
After any fix, re-run every criterion, including the ones that already passed. A change aimed at one criterion can break another, and the passing criteria are where nobody is looking. Had the agent re-run only criteria 1, 4 and 5 after round 2, it would have reported three fixes, and the regression in criterion 3 would have gone unnoticed. The cheap checks take seconds, so skipping them saves nothing.
One change per experiment
Round 2 made three changes and moved four criteria. Any of the three changes could have broken criterion 3, or two of them together, and the data cannot tell which. So go back to round 1 and make the changes one at a time, scoring every criterion after each.
| Round | Change | 1 | 2 | 3 | 4 | 5 | 6 | 7 | Score |
|---|---|---|---|---|---|---|---|---|---|
| 1 | First build | F | P | P | F | F | P | P | 4/8 |
| 2 | Index, stemming and fuzzy matching at once | P | P | F | P | P | P | P | 6/8 |
| 3 | From round 1: index only | P | P | P | F | F | P | P | 5/8 |
| 4 | Add stemming | P | P | P | F | P | P | P | 6/8 |
| 5 | Add fuzzy matching | P | P | F | P | P | P | P | 6/8 |

Rounds 4 and 5 have the same total. A log of totals would show round 5 as a change that did nothing. It fixed criterion 4 and broke criterion 3. Fuzzy matching lets "rice" match "rich" and "ice", those recipes have more saves, and so "Rich chocolate cake" and "Lemon ice cream" push two of the three rice dishes out of the top 3.
With the cause known, the fix can aim at it. Round 6 replaces round 5's change: fuzzy matching runs only when exact matching finds fewer than three recipes, so "rice", with three exact matches, never reaches it. Round 6 scores 7 of 8, with criterion 8 still pending.
The playground is a cut-down version of the same climb: a 14-recipe catalog, the stemming and fuzzy switches (the index only affects speed), and criteria 3, 4 and 5 as code. Each round is compared criterion by criterion with the round it started from.
type Recipe = { title: string; saves: number };
type Build = { name: string; from?: string; stemming: boolean; fuzzy: "off" | "always" | "fallback" };
const recipes: Recipe[] = [
{ title: "Rich chocolate cake", saves: 900 }, { title: "Chicken curry", saves: 820 },
{ title: "Roast potatoes", saves: 700 }, { title: "Banana bread", saves: 650 },
{ title: "Beef lasagna", saves: 610 }, { title: "Lemon ice cream", saves: 500 },
{ title: "Tomato soup", saves: 450 }, { title: "Sourdough bread", saves: 400 },
{ title: "Potato salad", saves: 350 }, { title: "Rice pudding", saves: 300 },
{ title: "Vegetable lasagna", saves: 260 }, { title: "Broccoli cheese bake", saves: 120 },
{ title: "Spiced rice", saves: 60 }, { title: "Coconut rice", saves: 40 },
];
const stem = (w: string) =>
w.endsWith("ies") ? w.slice(0, -3) + "y" : w.endsWith("oes") ? w.slice(0, -2) : w.endsWith("s") && !w.endsWith("ss") ? w.slice(0, -1) : w;
const words = (s: string, b: Build) =>
s.toLowerCase().split(/[^a-z]+/).filter(Boolean).map((w) => (b.stemming ? stem(w) : w));
// Edit distance between two words: insertions, deletions and substitutions.
function distance(a: string, b: string): number {
let prev = Array.from({ length: b.length + 1 }, (_, j) => j);
for (let i = 1; i <= a.length; i++) {
const row = [i];
for (let j = 1; j <= b.length; j++)
row[j] = Math.min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1));
prev = row;
}
return prev[b.length];
}
// More matching words ranks higher; ties go to the recipe with more saves.
function rank(query: string, b: Build, fuzzy: boolean): string[] {
const q = words(query, b);
const near = (w: string, t: string) => w === t || (fuzzy && w.length >= 4 && distance(w, t) === 1);
return recipes
.map((r) => ({ r, hits: q.filter((w) => words(r.title, b).some((t) => near(w, t))).length }))
.filter((x) => x.hits > 0)
.sort((x, y) => y.hits - x.hits || y.r.saves - x.r.saves)
.map((x) => x.r.title);
}
function search(query: string, b: Build): string[] {
const exact = rank(query, b, b.fuzzy === "always");
return b.fuzzy === "fallback" && exact.length < 3 ? rank(query, b, true) : exact;
}
// Criteria 3, 4 and 5 from the table, as code. Each returns true for pass.
const top3 = (q: string, b: Build) => search(q, b).slice(0, 3);
const criteria: Record<string, (b: Build) => boolean> = {
"3 reference": (b) =>
[
{ q: "rice", accept: ["Rice pudding", "Coconut rice", "Spiced rice"] },
{ q: "bread", accept: ["Banana bread", "Sourdough bread"] },
{ q: "lasagna", accept: ["Beef lasagna", "Vegetable lasagna"] },
].every(({ q, accept }) => top3(q, b).length > 0 && top3(q, b).every((t) => accept.includes(t))),
"4 typos": (b) =>
[["chiken", "Chicken curry"], ["brocoli", "Broccoli cheese bake"], ["lasagne", "Beef lasagna"]]
.every(([q, want]) => top3(q, b).includes(want)),
"5 plurals": (b) =>
[["tomato", "tomatoes"], ["potato", "potatoes"]]
.every(([one, many]) => top3(one, b).join() === top3(many, b).join()),
};
const rounds: Build[] = [
{ name: "first build", stemming: false, fuzzy: "off" },
{ name: "stemming + fuzzy at once", from: "first build", stemming: true, fuzzy: "always" },
{ name: "stemming only", from: "first build", stemming: true, fuzzy: "off" },
{ name: "+ fuzzy always", from: "stemming only", stemming: true, fuzzy: "always" },
{ name: "+ fuzzy as fallback", from: "stemming only", stemming: true, fuzzy: "fallback" },
];
// Score every criterion in every round, and compare each round with the one it started from.
const history = new Map<string, Record<string, boolean>>();
for (const b of rounds) {
const now = Object.fromEntries(Object.entries(criteria).map(([k, check]) => [k, check(b)]));
history.set(b.name, now);
const base = b.from ? history.get(b.from) : undefined;
const passed = Object.values(now).filter(Boolean).length;
const cells = Object.keys(now).map((k) => `${k} ${now[k] ? "pass" : "FAIL"}`).join(" | ");
const moved = base
? Object.keys(now).filter((k) => now[k] !== base[k]).map((k) => (now[k] ? `fixed ${k}` : `BROKE ${k}`)).join(", ")
: "";
console.log(`${b.name.padEnd(25)} ${passed}/3 ${cells}${moved ? ` <- ${moved}` : ""}`);
}
console.log("top 3 for rice, fuzzy always:", top3("rice", rounds[3]).join(", "));Output
The second line is round 2 in miniature: two criteria fixed, one broken, total up. The fourth line keeps the third line's total while two criteria trade places, and the last line shows why. Try making an exact match count for more than a fuzzy one inside rank, and see whether "fuzzy always" still breaks criterion 3.
A change here means one idea whose effect you want to know. The index is one idea and stemming is another. Two edits that only make sense together, like a new column and the code that fills it, count as one. When you cannot say which of several edits did what, split them and score each.
Slow hills
Criteria 1 to 7 move in seconds, and an afternoon is enough for a dozen experiments. Criterion 8 moves in weeks, because every measurement needs real people using the live site. A climb whose criteria need real-world data is a slow hill, and it needs different habits.
The search launches. These are the first three readings of criterion 8, with the target at 45%:
| Window | Searches | Top-3 click rate | 95% interval, roughly | Result |
|---|---|---|---|---|
| Day 1 | 410 | 38% | 4.7 points either way | Pending |
| Week 1 | 4,300 | 41% | 1.5 points either way | Fail |
| Week 2, with a photo and cooking time in each result | 4,500 | 49% | 1.5 points either way | Pass |
After day 1 the agent proposed a ranking change. One day is too little data to act on, even though the simple interval already sits below 45%: searches by one person in one session are related, so the real uncertainty is wider than that interval, and a single day misses the weekly cycle of how people cook. The criterion already says so by asking for 7 days and 2,000 searches, which makes day 1 pending.
Week 1 is a fail you can trust, since its whole interval sits below 45%. Pick one change for the next week. Every result showed a title and nothing else, so a cook could not tell three lasagnas apart, and week 2 added a photo and the cooking time.
Two different weeks still differ in other ways, such as a holiday or a newsletter that brought new visitors. The stronger experiment, an A/B test, shows each layout to half the visitors in the same week. Meanwhile, keep climbing the fast criteria, and leave alone anything the slow one depends on.
When the criteria should change
Criteria are written before the build exists, by a person who keeps learning as it goes. Some of what they learn belongs in the criteria. The skill is telling a real lesson apart from pressure to make the score look good.
Stuck at round 5, the agent offered another way out: loosen criterion 3 from all 40 reference queries to 30 of 40, "since fuzzy matching adds some noise." That proposal comes entirely from what the current build does, and nothing about what cooks want has changed. Reject it and fix the build, as round 6 did. Deleting a failing criterion to report a perfect score, with no dated reason and no evidence the original target was wrong, hides the failure from everyone who reads the score.

A week into production, the search logs show that 10 of the 40 reference queries were never typed by anyone, while "air fryer" and "quick dinner" are among the top 20 real queries. Replacing those 10 with real queries is a sound change to the criteria, because it comes from evidence about what users want, even if the current build fails the new set.
One question separates the two cases: would you have written the new criterion before you saw this build? If yes, change the criterion. If it exists only because the build cannot meet the old one, change the build. Either way, record the change in the ideal state's decisions log with the date and the reason, and re-score the current build against the new criteria.
Passing everything and still wrong
At 8 of 8, a tester signs in with a second account and searches "curry". The third result is "Grandma's curry", a recipe another user marked private. Every criterion passes, and the search is still wrong.
Daniel Miessler names this signal in "General Hill-Climbing": "The best way to tell if you need to update your evals is if you get a result that perfectly passes your evals, but you still don't like it for some reason." A result like this means a criterion is missing. The eight criteria describe good results and say nothing about whose recipes may appear in them.

Write the missing criterion so a stranger could check it: for each of 20 test accounts, no search returns a recipe its owner marked private. Run it against the current build before anyone touches the code. It fails, and the score drops to 8 of 9. That drop is progress: the criteria are sharper, and the failing run proves the new check can see the problem. The agent filters results by visibility in the query, and the score returns to 9 of 9.
A fix with no criterion leaves the next change free to leak private recipes under a perfect score. Accepting the build on its score, when you can see it is wrong in use, is the same mistake one step earlier.
Keeping the score history
The record of a climb is a log with one line per round: what changed, the result for each criterion, and the score. Changes to the criteria get their own lines, with the reason in the decisions log.
round change 1 2 3 4 5 6 7 8 9 score
1 first build F P P F F P P - 4/8
2 index + stemming + fuzzy (agent, at once) P P F P P P P - 6/8
3 from 1: index only P P P F F P P - 5/8
4 + stemming P P P F P P P - 6/8
5 + fuzzy matching P P F P P P P - 6/8
6 fuzzy as fallback only (replaces 5) P P P P P P P - 7/8
7 launch, week 1 P P P P P P P F 7/8
8 photo and cooking time in each result P P P P P P P P 8/8
9 criterion 3 revised: 10 queries from logs P P P P P P P P 8/8
10 criterion 9 added: no private recipes P P P P P P P P F 8/9
11 filter results by visibility P P P P P P P P P 9/9A row shows what one change did. A column shows when a criterion first passed and whether it ever broke again. The totals alone hide the regression in round 5, and they make round 10 look like a step backward when it was the criteria getting sharper.
A climb you can trust changes one thing per round, re-scores every criterion and writes the result down. Its criteria change only for a reason you would have accepted before seeing the build, and a perfect score on a build you can see is wrong means the next thing to write is a criterion.