Building Software
Sections

Contents Section 9, Delegating to Agents

Reading What Agents Built: Understanding and Risk-Based Review

Reading agent-built code to understand what was built and why, and aiming human review where criteria and evals cannot reach: security, money, one-way doors and any behavior no criterion covers.

Section 9 of 9, subsection 4 of 6. Depth: do. Core 24 of 25. A 9-minute read.

Mistakes to catch in review

  1. A change that passes every criterion but alters behavior outside the requested task, such as a shared helper whose new default changes every existing caller.

  2. A feature merged on passing checks that nobody on the team can explain: asked what happens when its job runs twice, the owner has to ask an agent.

  3. A report that says all criteria pass with no command, output or commit attached, or with a run made before the agent's final edit.

  4. Line-by-line attention spent on template markup and test setup while the few lines that send email, move money or delete data go unread.

  5. An agent reviewer approving another agent's change because both read the same description and share its wrong assumption.

Reading What the Agent Built

Your agent finished overdue-invoice reminders for your billing app. Before it started, you wrote the ideal state for the feature and four criteria a stranger could check:

Markdown
## Reminders
A customer whose invoice is past due hears about it by email, so overdue
invoices get paid without anyone on the team chasing them by hand.

- R1: An open invoice past its due date gets a reminder email that names
  the invoice number and the amount due.
- R2: A paid or void invoice gets no reminder.
- R3: The reminder job runs every hour.
- R4: The email reads as polite and clear, judged against three sample
  emails the support lead wrote.

The agent's report lists each criterion, the command that checked it and the result: four passes on commit 7c1e0b2. The change is about 500 lines across seven files, and CI has not finished on the branch yet.

Reading all 500 lines would take an afternoon, and the four checks already prove most of what they cover. Two questions are left, and they decide where your reading goes: which behavior nobody wrote a criterion for, and whether you understand what was built well enough to run it, secure it and change it next month.

What reading is for

Code review used to be a gate: a person approved each diff before it merged. When agents write most of the code, a person in the loop on every step becomes the bottleneck, and a check that runs on every commit proves its criterion more reliably than a tired reader skimming the same lines. The criteria and their checks carry the routine proof, and human reading keeps two jobs that checks cannot do.

The first job is understanding. In his 1985 essay "Programming as Theory Building", Peter Naur held that the real program is the understanding its programmers carry: what each part is for, why it has this shape, how it would bend to a new need. The code records only part of it, so with agent-written code someone has to build that understanding on purpose. The line between engineering and vibe coding sits here: whether you understand what was built and why, whoever typed it.

The second job is risk. Some behavior has no criterion, because nobody thought of it, and some mistakes cost so much that a passing check is thin evidence. Security, money and one-way doors (actions you cannot take back: a sent email, a deleted row, a migration) get human reading even when every check is green, and so does any behavior no criterion mentions.

Read the checks before the code

Read the tests behind each criterion before the implementation. They show what the agent decided "done" means, and reading them first keeps the implementation from persuading you. For each check, ask what wrong version of the code would still pass it.

The check for R1 creates one open invoice due yesterday, calls sendOverdueReminders(now) once, and asserts that exactly one email was sent. Here are wrong versions it lets through:

  • A job that sends the same reminder every time it runs. The test calls it once.
  • A job that treats an invoice due later today as overdue. Every fixture is at least a day overdue.
  • Dates formatted in the wrong time zone. The suite runs with TZ=UTC and every fixture date is midnight UTC.
  • A job that stops at the first failed send and skips everyone after it. The fake mailer always succeeds.

Every item on that list is behavior the criteria leave open. The ideal state says a customer "hears about it by email" and stops there: how often, and what happens when one address bounces, are unwritten. The list is your reading plan for the implementation.

Build the theory from the entry point

To understand code you did not write, find where it starts, follow the data, and write down every side effect. Side effects are the actions that reach outside the function: database writes, network calls, files, emails.

The entry point is in schedule.ts:

TypeScript
scheduler.every("1 hour", () => sendOverdueReminders(new Date()));

The job itself, in job.ts:

TypeScript
export async function sendOverdueReminders(now: Date): Promise<void> {
  const invoices = await db.invoices.findMany({
    where: { status: "open", dueAt: { lt: now } },
  });
  for (const inv of invoices) {
    await email.send({
      to: inv.customer.email,
      subject: `Invoice ${inv.number} is overdue`,
      body: renderReminder(inv),
    });
  }
}

Every hour, the query selects every open invoice past its due date, and the job emails each one. Sending the email is the only side effect, and nothing records that a reminder went out. Write the theory down in plain sentences, the way you would explain it to whoever is on call next week:

Output
Every hour the scheduler calls sendOverdueReminders(now).
It selects every open invoice whose dueAt is before now and emails each one.
Side effect: one email per selected invoice. Nothing is written back.
So an invoice that stays open is selected, and emailed, on every run.
A send that throws ends the loop; every invoice after it is skipped until the next run.
A scheduler set to `"1 hour"` runs `sendOverdueReminders` every hour, which selects from a query of open, past-due invoices. Each invoice goes to the highlighted `email.send`, the only side effect. From `email.send` one arrow, labeled nothing recorded, leads to 72 reminders for one invoice over 72 hourly runs; another, labeled throws, leads to later invoices skipped. A note reads: the test called it once.
Every hour the query selects every open invoice past its due date, and email.send is the only side effect, so with nothing recorded one invoice gets 72 reminders over 72 hourly runs. If one send throws, the job exits and every later invoice is skipped.

The last two lines are predictions, and they come from the trace alone. Ask what happens when the job runs twice: the same invoices come back, and the same customers get a second email. For one invoice that stays open through 72 successful hourly runs, the job sends 72 reminders. If the query keeps returning a bad address early in its results, the customers behind it get no reminder at all. R1 through R4 still pass, and both behaviors reach real customers.

The theory also says where a change would go. If the product owner asks for "remind again after seven days", you need a recorded send time and a query that reads it. Being able to say that without asking the agent is the test of whether you understand what was built.

Turn the predictions into checks

A prediction you can run settles the question faster than one you argue about. The playground models the agent's job and a version that records each reminder and keeps going after a failed send. It runs three checks against both: the agent's R1 test, and two checks written from your theory.

TypeScriptRuns in your browser
type Invoice = { number: string; email: string; dueAt: Date; reminderSentAt: Date | null };
type Job = (db: Invoice[], now: Date) => Promise<void>;

// In-memory stand-ins for the invoices table and the email provider.
const due = new Date("2026-10-01T00:00:00Z");
const HOUR = 3_600_000;
const at = (hours: number) => new Date(due.getTime() + hours * HOUR);
const invoice = (number: string, email: string): Invoice => ({ number, email, dueAt: due, reminderSentAt: null });
let sent: string[] = [];
let failed: string[] = [];
async function send(to: string): Promise<void> {
  if (!to.includes("@")) throw new Error(`rejected address ${to}`);
  sent.push(to);
}

// The agent's job, as traced: every open invoice past its due date gets an email.
const agentJob: Job = async (db, now) => {
  for (const inv of db.filter((i) => i.dueAt < now)) await send(inv.email);
};

// Records each reminder and keeps going after a failed send.
// This fixes sequential runs only; overlapping runs still need an atomic claim.
const recordedJob: Job = async (db, now) => {
  for (const inv of db.filter((i) => i.dueAt < now && i.reminderSentAt === null)) {
    try {
      await send(inv.email);
      inv.reminderSentAt = now;
    } catch (e) {
      failed.push(`${inv.number}: ${(e as Error).message}`);
    }
  }
};

// Each check returns true when the job does what the check expects.
const checks: [string, (job: Job) => Promise<boolean>][] = [
  ["agent's R1 test: one run, one email", async (job) => {
    await job([invoice("INV-1", "ana@example.com")], at(1));
    return sent.length === 1;
  }],
  ["three hourly runs: still one email", async (job) => {
    const db = [invoice("INV-1", "ana@example.com")];
    for (let h = 1; h <= 3; h++) await job(db, at(h));
    return sent.length === 1;
  }],
  ["a bad address: the next customer still hears", async (job) => {
    const db = [invoice("INV-1", "ana@example.com"), invoice("INV-2", "not-an-address"), invoice("INV-3", "kim@example.com")];
    await job(db, at(1)).catch(() => {});
    return sent.includes("kim@example.com");
  }],
];

for (const [name, job] of [["agent's job", agentJob], ["recorded job", recordedJob]] as const) {
  for (const [label, check] of checks) {
    sent = [];
    failed = [];
    console.log(`${name.padEnd(13)} ${(await check(job)) ? "pass" : "FAIL"}  ${label}`);
  }
}

The agent's job passes its own test and fails both checks your theory produced. The recorded job passes all three. Recording each reminder fixes sequential runs only. Overlapping runs and a crash between the send and the record can still duplicate an email, so the full fix also claims each invoice atomically before sending and passes an idempotency key to the email provider where it supports one.

What you do next goes through the ideal state. Add the missing criteria, such as "R5: an invoice gets at most one reminder in any seven days" and "R6: a failed send for one invoice still lets every other reminder go out", each with its check, and hand them back to the agent. The agent writes the fix, and it lands against checks that will catch the same mistake in every later version.

Where criteria cannot reach

With the checks read and the theory written, rank what is left by two things: whether a check covers it, and whether a mistake there is cheap to undo or expensive or one-way.

A two-by-two grid. The columns read cheap to undo, and expensive or one-way. The rows read check covers it, and no check covers it. Covered and cheap: skim, `template.ts` under R4. Covered and expensive: read the check, `job.ts` under R1. Not covered and cheap: add a criterion, when you have time. Not covered and expensive, highlighted: line by line, `money.ts` and `formatDate.ts`. A note under the grid reads: read where checks stop.
Every part of the change lands in one cell, set by whether a check covers it and whether a mistake there is cheap to undo. Line-by-line reading goes to the bottom-right cell, where no check covers an expensive or one-way mistake, and to any covered part whose check turns out weak.

A part a check covers, where a mistake is cheap to undo, gets a skim; the email's wording under R4 is one, and you read the rendered email. A part a check covers, where a mistake is expensive, gets its check read closely, and every line too if the check turns out weak, because a weak check there gives false confidence. A part no check covers, where a mistake is cheap, earns a new criterion when you have time. A part no check covers, where a mistake is expensive or one-way, gets read line by line now.

Part Covered by Cost of a mistake How to read it
job.ts query and send loop R1, weakly Repeated or missing emails; a sent email cannot be recalled Every line
money.ts default parameter No criterion Wrong amounts shown across the product Every line, plus every caller
formatDate.ts No criterion Wrong due date in customer emails Every line
template.ts wording and layout R4 Awkward copy, fixed next week Skim, then read the rendered email
schedule.ts registration R3 Job runs at the wrong interval Every line; it is three lines

About 40 of the 500 lines decide whether the feature is safe. Most of the rest is template markup and test setup that the checks already exercise, and spending your line-by-line attention there while the send loop goes unread is the expensive mistake.

Security sits in the expensive column every time. This change touches no permission check, no secret and no input that reaches a query or a shell. If it did, that hunk would be read line by line whatever the checks say, because a missing authorization check passes every test that logs in as the owner.

The code the diff does not show

money.ts is outside the task, so none of the four criteria can see it. Here is what changed there, and the snapshot the agent updated:

Diff
-export function formatMoney(amount: number, currency: string): string {
+export function formatMoney(amount: number, currency: string, showCurrency = false): string {
   const n = amount.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2 });
-  return `${currency} ${n}`;
+  return showCurrency ? `${currency} ${n}` : n;
 }
Diff
-    <td class="total">USD 1,250.00</td>
+    <td class="total">1,250.00</td>
Five steps in a chain. `showCurrency` defaults to `false`, highlighted, leading to the invoice page losing its currency code, then the snapshot test failing, then the agent updating the snapshot, then tests pass. Beside tests pass, large text reads: the test changed to agree.
The new showCurrency parameter defaults to false, so the invoice page loses its currency code and its snapshot test fails. The agent updated the snapshot to match, and the tests pass because the test was changed to agree with the new behavior.

The agent wanted amounts without a currency code in its email template and added a parameter to get them. The default is false, so every existing caller that passes two arguments now gets the bare number. The invoice page lost its currency code, its snapshot test failed, and the agent updated the snapshot to match. The tests pass because the test was changed to agree with the new behavior.

A changed test for code outside the task is a question to settle first: did the task ask for this output to change? Here it did not. A diff shows the lines that changed and leaves out the lines that depend on them, so search for the callers. The results include the definition in src/lib/money.ts; the caller matches are excerpted here:

Output
$ git grep -n "formatMoney(" -- src
src/invoices/page.ts:41:      formatMoney(inv.total, inv.currency)
src/invoices/pdf.ts:88:       formatMoney(line.amount, inv.currency)
src/billing/receipt.ts:23:    formatMoney(payment.amount, payment.currency)
src/reminders/template.ts:30: formatMoney(inv.total, inv.currency, false)

Three callers outside the task changed behavior, and one of them is covered by the snapshot the agent rewrote. The fix is a default of true, or a separate function for the email. Say so in your reply to the agent, and add a criterion to the invoice page's ideal state that every amount it shows carries its currency.

The same search finds duplicates. The new formatDate.ts is a second date-formatting helper next to the existing one, with different time-zone handling. It formats with toISOString, which always gives the UTC date, while the invoice page uses formatInZone(d, timeZone) from src/lib/dates.ts, which gives the date on the customer's own calendar. For a customer in Tokyo whose invoice is due at local midnight on 15 October, the stored instant is 2026-10-14T15:00:00Z: the invoice page says the 15th and the reminder says the 14th. When a change adds a small utility for dates, money, strings, retries or paths, look for the one the project already has. Deleted lines deserve the same attention as added ones, since a removed check or a removed await changes behavior as much as new code does.

Claims and their evidence

The report says all four criteria pass on commit 7c1e0b2. git rev-parse --short HEAD prints 9f3a4d1. The agent's transcript shows one more edit after its run: it reworded a sentence in the email body. Run the checks on the head commit and R4's judge still passes, but one test in job.test.ts fails, because it checks for the old sentence.

The claim was true of an earlier commit and was carried forward into the report unchanged. Treat a claim that something passes as evidence only when it names the command, shows the output and ran on the exact commit you are looking at. A green CI run on that commit qualifies, and so does a run you start yourself.

Model reviewers share the blind spot

It is tempting to ask a second agent to review the first agent's work. A model reviewer adds a pass over all 500 lines and may catch typos, unused imports or a missing null check. Weigh each of its findings; its approval alone is weak evidence.

The description, hiding the currency for email, and the diff, which matches that sentence, both point to the highlighted model reviewer, to which the change looks correct. The existing callers sit outside the diff; a dashed arrow from the reviewer reaches them only if it searches.
The model reviewer reads the same description and the same diff, and the diff matches that sentence, so the change looks correct. The callers that broke sit outside the diff, and it sees them only if it searches for them.

It is less likely to catch the defects above. The reviewing model reads the same diff and the same description. The description says formatMoney "now supports hiding the currency for email templates", and the diff matches that sentence, so the change looks intentional and correct. The callers that broke sit outside the diff, and the reviewer sees them only if it searches for them. When two models accept the same assumption from the same description, the second approval adds little evidence, and a reviewer from another vendor handed that description can share the assumption too.

Give a model reviewer a specific search, such as every caller of each changed shared function, and confirm from its report that it ran it. Do the reading that builds the theory yourself, because an agent's summary of the code is one more description, and the theory has to live in the head of the person who will run the system and change it.

Check your understanding

8 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An agent finished "Add a discount banner to checkout", and every criterion for the banner passes with evidence on the head commit. The change also edits this shared helper, and the banner calls roundPrice(amount, "floor"). Where does your reading go first?
    Diff
    -export function roundPrice(cents: number): number {
    -  return Math.round(cents);
    +export function roundPrice(cents: number, mode: "round" | "floor" = "floor"): number {
    +  return mode === "floor" ? Math.floor(cents) : Math.round(cents);
     }
    Show the answer

    Answer C. The new parameter defaults to floor, so every caller that passes one argument changed behavior. None of those callers appear in the diff and no banner criterion mentions them, so no passing check speaks for them. The banner's floor mode is the part the criteria already cover.

  2. 2This is the only check behind the criterion "a 10% coupon takes 10% off the cart". Which wrong implementation of total still passes it?
    TypeScript
    test("applies a 10% coupon", () => {
      const cart = { items: [{ priceCents: 1000, qty: 1 }], coupon: "SAVE10" };
      expect(total(cart)).toBe(900);
    });
    Show the answer

    Answer B. With a single item at 1000 cents, 10% off and a flat 100 cents off both give 900, so the check cannot tell them apart. The other three return 1000, 810 and 9, and fail. Asking what wrong version would pass shows where the check stops, and which check to add: a second cart at a different price.

  3. 3An agent's report says all five criteria pass on commit a1b2c3d. The head of the branch is e4f5a6b: after the run, the agent renamed a function in two files. What do you do with the report?
    Show the answer

    Answer D. The green run describes a commit that is no longer the head, so it says nothing about the code in front of you. A rename can miss a caller or a string reference and break the build. A pass counts as evidence only with the command, its output and the exact commit it ran on, and that commit has to be the head.

  4. 4The description says parseAmount "now also accepts European formats like 1.250,00", and a model reviewer approved the change. What would give the model reviewer's pass independent value?
    Show the answer

    Answer A. The risk is outside the diff: a caller that passes 1.250 meaning one and a quarter may now get 1250. A reviewer handed the same description shares its assumption that only new formats are affected, whichever vendor made it. A specific search over the callers brings evidence the description cannot supply, and its report shows whether the search happened.

  5. 5A 400-line agent change has passing checks, with evidence on the head commit, for its three criteria: the settings page renders, the new toggle saves, and the nightly cleanup still runs. Which part gets line-by-line reading?
    Show the answer

    Answer C. "The nightly cleanup still runs" is silent on which accounts it deletes, so no check reaches the line that matters, and a wrong condition there destroys customer data with no undo. The CSS sits under the render check and is cheap to fix, and a missed rename usually fails the build. Size is a poor guide: the six lines carry more risk than the 250.

  6. 6You are building your theory of an agent-written function by tracing its side effects. If billing.cancelSubscription throws, what state is left behind?
    TypeScript
    async function closeAccount(id: string): Promise<void> {
      await db.accounts.update({ where: { id }, data: { status: "closed" } });
      await billing.cancelSubscription(id);
      await email.send({ to: await lookupEmail(id), template: "goodbye" });
    }
    Show the answer

    Answer B. Each await is a separate side effect, and nothing wraps them in a transaction, so the account write stays done when the billing call throws, and the email line never runs. The throw does not say whether billing cancelled: the request may have failed, or succeeded with the response lost. Rollback happens only when code asks for it; an exception passing through a function leaves finished writes in place.

  7. 7An agent's change is titled "Speed up project deletion", and its one criterion, that deleting a project takes under 200 ms, passes. Here is one hunk. What is the problem?
    Diff
     export async function deleteProject(user: User, projectId: string): Promise<void> {
    -  if (!(await canAdmin(user, projectId))) throw new Forbidden();
       await db.projects.delete({ where: { id: projectId } });
     }
    Show the answer

    Answer D. The removed line was the authorization check, so the speed-up came from deleting a security control, and a timing criterion has no way to notice. Deleted lines change behavior as much as added ones and are easy to skim past. Caching the check would be a fair speed-up proposal, but this diff drops the check outright.

  8. 8Over a month, a teammate merged 30 agent changes to the refunds service, each on passing criteria with evidence attached. Support asks her what happens when a refund is issued twice for the same payment, and she says she would have to ask the agent. What closes the gap?
    Show the answer

    Answer A. The checks proved what their criteria said, and none of the criteria covered a repeated refund; what is missing is her theory of the service. Approving every diff is slower and still has her reading changes one at a time, which seldom adds up to knowing the whole path. Agent-written documentation is one more description to trust. Tracing the path from its entry point and writing down what it does is the theory, and the prediction it yields becomes a new criterion with its check. A criterion that refunds must be correct gives no check anything to test.

You understand it when you can

Drill

An agent finished 'Add CSV export' against four criteria and reports all four passing on the head commit. Its 600-line change also edits the shared date-formatting helper, adds a second date helper with different time-zone handling, and updates two snapshot tests to match. Without asking an agent, write the export's theory in a few sentences (entry point, data, side effects), mark which parts the four checks already prove, mark the lines no criterion reaches that deserve line-by-line reading, and write the criteria you would add.

Topics

Building the Theory of the System
Tracing entry points, data flow and side effects until you can explain what was built and why, predict what it does in a case nobody tested, and say where a change would go.
Reading the Checks First
Asking what wrong implementation each test or eval would still pass, which shows where the criteria reach and where they stop.
Risk-Based Review
Spending human reading where criteria cannot reach: security, money, one-way doors and any behavior no criterion covers, ranked by what a mistake costs and whether it can be undone.
Reading Diffs
Reading a change in the context of the code around it: callers outside the diff, deleted lines, rewritten tests and helpers that duplicate one the project already has.
Evidence for Claims
Accepting that something passes only with the command, its output and the exact commit it ran on.
Automated Reviewers and Shared Blind Spots
Giving model reviewers specific searches while remembering that two models handed the same description can be wrong in the same way.

Further reading

Start here

  • Google Engineering Practices: How to do a code review. Reference.

    Google's published reviewer guide sets the order to look at a change (design, functionality, complexity, tests, then naming), says to read every line, and asks reviewers to put the most attention on the parts that carry the most risk.

Short reads

Watch

Books

  • The Programmer's Brain: What every programmer needs to know about cognition. Felienne Hermans, 2021.

    Uses research on working memory and chunking to teach concrete methods for reading unfamiliar code and diffs quickly without losing the thread.

  • Looks Good to Me: Constructive Code Reviews. Adrienne Braganza, 2025.

    A full review process: what to check and in what order, how to tell load-bearing lines from routine ones, and when to reject a change instead of approving it.

  • Working Effectively with Legacy Code. Michael C. Feathers, 2004.

    The classic on understanding code you did not write by finding its seams and pinning its behavior with characterization tests, which is how you check whether an agent's change altered behavior outside the task.