Building Software
Sections

Contents

Section 2 of 9

Verification and Evals

Proving You Got What You Wanted

6 subsections, the core path through it Core 3–7.

Why it matters when agents do the typing

Intent lives in the written criteria of the ideal state, and tests and evals are their executable form: each criterion carries the cheapest check that could prove it false, from an exact assert to a calibrated judge to a number from production. A check gives the same verdict on every run and does not care who is asking, so it keeps measuring long after the conversation that produced the code is gone, and the code can be thrown away and rebuilt against the same checks. When the same process writes both the code and its checks, both can carry the same misunderstanding, so the engineer owns what the checks assert, scores every build against them, and sharpens the criteria whenever a build passes them and is still wrong.

The course: 6 lessons, about 55 minutes of reading, in the order below.

  1. 2.1

    Tests as Executable Specifications

    Core 3 · Depth: do

    Test-first thinking without the ritual: every behavior has a test, each test comes from a requirement, and nothing is done until the tests say so.

    A mistake it teaches you to catch: A failing test 'fixed' by changing its expected value to match the buggy output.

    Lesson: Where the Expected Value Comes From 9 min

  2. 2.2

    Engineering Robust Tests

    Core 4 · Depth: do

    Choosing the right level of test, controlling time and randomness, and building suites that stay fast, deterministic and trustworthy.

    A mistake it teaches you to catch: Everything mocked, so the tests never touch a real database and miss every query bug.

    Lesson: A Test Should Give the Same Answer Every Time 8 min

  3. 2.3

    Property-Based Testing and Test Quality

    Depth: do

    Checking invariants across thousands of generated inputs, and measuring whether your tests would actually catch a bug.

    A mistake it teaches you to catch: Code that passes every hand-written example and fails on an input nobody thought of, such as an empty string, a huge number or combining characters.

    Lesson: Let the Computer Choose the Inputs 8 min

  4. 2.4

    Continuous Testing and Fast Feedback

    Core 5 · Depth: do

    Running every check on every change so you know within minutes, and ideally seconds, when something breaks.

    A mistake it teaches you to catch: A failing test marked as skipped so the pipeline turns green.

    Lesson: What a Green Check Proves 10 min

  5. 2.5

    Evals: Measuring Any Output Against Its Criteria

    Core 6 · Depth: do

    Measuring any output an agent produces, from a model's reply to a report or a page, against its written criteria with the cheapest check that works, so a change is judged on evidence.

    A mistake it teaches you to catch: A change shipped because its output looked better on three hand-picked examples.

    Lesson: Evidence That an Output Is Good 10 min

  6. 2.6

    The Loop: Build, Evaluate, Iterate

    Core 7 · Depth: do

    Scoring a build against its written criteria, fixing what failed, scoring everything again, and sharpening the criteria themselves, one change at a time, until the build is what you wanted.

    A mistake it teaches you to catch: An agent changes several things in one round, the total score rises, and nobody can say which change helped and which one broke a criterion that used to pass.

    Lesson: Score It, Change One Thing, Score It Again 10 min

Section test: Verification and Evals

11 questions across the 6 lessons above. Take it after the course; each answer explains itself.

  1. 1An agent's pull request, titled update snapshots after refactor, changes this file, and CI goes green. The requirement says SAVE10 takes 10% off the cart. What does the green check tell you?
    Diff
      // __snapshots__/invoice.test.ts.snap
      exports[`invoice for 3 items at $30.00 with SAVE10 1`] = `
    - "Total: $81.00"
    + "Total: $90.00"
      `;
    Show the answer

    Answer C. Running the tests with -u copies whatever the code prints into the snapshot, so the new expected value came from the code under test, and CI then compares the code with its own output. Work it from the requirement: 3 × $30.00 is $90.00, and 10% off gives $81.00, so the refactor stopped applying the discount. CI does not rewrite snapshots unless someone passes the update flag; it compares against the committed file, which is why this diff is the thing to review.

  2. 2The requirement says a card surcharge is 2% of the amount, rounded to the nearest cent with halves up. An agent's pull request, titled fix float flakiness in surcharge test, makes this change. Which wrong result does the test now accept?
    Diff
      test("2% surcharge, rounded to the nearest cent", () => {
    -   expect(surcharge(37525)).toBe(751);
    +   expect(surcharge(37525)).toBeCloseTo(751, -1);
      });
    Show the answer

    Answer A. With -1 digits, toBeCloseTo passes when the difference from 751 is under 5, so anything from 747 to 755 passes, including the 750 that a truncating Math.floor returns. NaN is close to nothing, so it still fails. The old expected value came from the requirement: 37525 × 2% is 750.5, which rounds up to 751. Nothing here was flaky, since the same input gives the same output on every run, so the change hid a failure.

  3. 3An agent's pull request, titled deflake checkout e2e, makes these two changes. Which bug can now merge with a green check?
    Diff
      // playwright.config.ts
    +   retries: process.env.CI ? 3 : 0,
    
      // checkout.spec.ts
        await page.click("#pay");
        await page.waitForTimeout(2000);
        expect(await page.textContent("#status")).toBe("Paid");
    Show the answer

    Answer C. With three retries, a test passes if any of four attempts passes, so a bug that fails one run in three gets through all but about 1.2% of the time, (1/3)^4. A bug that fails every run fails every attempt, including the 5-second payment, which misses the fixed 2-second sleep each time. The flakiness came from that sleep, a guess about timing. Waiting for the status text with a deadline removes it without hiding real failures.

  4. 4The requirement says shipping costs 4.5% of the order, rounded to the nearest cent with halves up. shippingFee truncates instead. An agent adds this property. What happens when it runs?
    TypeScript
    test("shipping fee matches the formula", () => {
      fc.assert(
        fc.property(fc.integer({ min: 0, max: 10_000_000 }), (orderCents) => {
          const expected = Math.floor(orderCents * 0.045);
          expect(shippingFee(orderCents)).toBe(expected);
        }),
      );
    });
    Show the answer

    Answer D. The expected value is computed with the same Math.floor the code uses, so the property checks the code against a copy of itself and cannot see the bug, however many inputs it draws. A reference comparison helps only when the reference comes from the requirement. With Math.floor((orderCents * 45 + 500) / 1000), which rounds halves up in integers, the property fails within its first few inputs. The first option describes that version, not this one.

  5. 5The tag round-trip property kept failing on [""], so the agent added the filter below and the test went green. What should a reviewer say?
    TypeScript
    const encodeTags = (tags: string[]) => tags.join("\n");
    const decodeTags = (s: string) => (s === "" ? [] : s.split("\n"));
    
    fc.assert(
      fc.property(
        fc.array(fc.string()).filter((tags) => !(tags.length === 1 && tags[0] === "")),
        (tags) => expect(decodeTags(encodeTags(tags))).toEqual(tags),
      ),
    );
    Show the answer

    Answer B. The filter bends the test to fit the code, the same move as editing an expected value until it passes. [] and [""] encode to the same string, so the decoder cannot tell them apart and one of them is lost. If the requirement forbids empty tags, encodeTags should reject them and the filter states that rule; if it allows them, the format has to change. A binary-unit string can be empty, and the default strings never contain a newline, so this property also never tries a tag with one.

  6. 6The integration test for trialsEndingSoon sits inside describe.skipIf(!process.env.DATABASE_URL). CI runs the job below, and a pull request swaps the bounds in the query. What does the pull request show?
    YAML
    integration:
      runs-on: ubuntu-latest
      services:
        postgres:
          image: postgres:16
          env:
            POSTGRES_PASSWORD: test
          ports: ["5432:5432"]
      steps:
        - uses: actions/checkout@v4
        - run: npm ci
        - run: npm run test:integration
    Show the answer

    Answer A. The job starts Postgres but never sets DATABASE_URL, so skipIf skips the only test that runs the SQL, the job exits 0 with one test skipped, and the swapped bounds merge. A real database is the right level for a query bug, but only if the test reaches it. Set DATABASE_URL in the step's env, and make the test fail when the variable is missing in CI instead of skipping quietly.

  7. 7This probe runs every 10 minutes against production and stayed green all week. On Thursday a config change in production disabled the search index, and since then every search returns HTTP 200 with an empty results list. Why did the probe not fail, and what fixes it?
    TypeScript
    // Criterion: a search for "rain jacket" returns at least one product.
    const res = await fetch("https://shop.example.com/api/search?q=rain+jacket");
    if (!res.ok) throw new Error(`search probe failed: status ${res.status}`);
    Show the answer

    Answer D. The probe is named after a criterion but checks something weaker: res.ok is true for any 2xx status, and a broken search still answers 200. A probe has to compare what comes back with the criterion, here that results holds at least one product, or it reports the server as up while the promise is broken. Running it more often repeats the same weak check. CI checks commits, and no commit changed on Thursday, so a rerun on main passes again.

  8. 8An agent built a sign-up screen and wrote an eval for it, choosing a check for each criterion. Which choice should change?
    Output
    criterion                                              check
    1  The form submits using the keyboard alone           judge, rubric on a screenshot
    2  Each error message says how to fix the input        question set, one question per message
    3  The screen looks like the rest of the product       judge, rubric on screenshots
    4  A password under 12 characters is refused           code assert
    Show the answer

    Answer C. Whether the form submits from the keyboard alone is a literal fact: a script can tab through the fields, press Enter and see whether the form was sent. A judge looking at a still image cannot see keystrokes, so its verdict is a guess, and it costs a model call on every run. Error messages suit narrow yes-or-no questions, because code cannot tell whether a message says how to fix the input. Looking like one design is taste, the case a judge is for, and a screen can match every color in the style sheet and still look out of place.

  9. 9A pull request is titled "Improve refund prompt; eval now 20/20". The extractor's task description says it must handle refund requests written in Spanish. Besides the prompt, the pull request contains this hunk in the eval dataset. What does the hunk do?
    Diff
     { "id": "spanish-refund",
       "email": "Quiero un reembolso del pedido A-20931, llegó roto.",
    -  "expected": { "intent": "refund", "orderId": "A-20931", "amountCents": null } }
    +  "expected": { "intent": "other", "orderId": "A-20931", "amountCents": null } }
    Show the answer

    Answer A. The email asks for a refund in Spanish, and the original label said so. Changing the expected answer after seeing the new prompt's output is the eval version of updating a snapshot to agree with a bug: the check passes because it was changed. Expected outcomes are written before the comparison and then frozen. In review, any change to a dataset or expected output inside the pull request it grades is a question: did the task ask for this answer to change?

  10. 10Criterion 8 reads: in production, at least 45% of searches end in a click on a top-3 result, over 7 days and at least 2,000 searches. The agent's climb log after launch day shows this line. What should the cell for criterion 8 say?
    Output
    round  change           1 2 3 4 5 6 7 8   score
    7      launch, day 1    P P P P P P P P   8/8   (8: 52% of 380 searches)
    Show the answer

    Answer C. The criterion names its own evidence, 7 days and 2,000 searches, and one day of 380 searches is not that evidence, so the result is pending and the score is 7 of 8 with one pending. A rate from one day also has a wide interval, about 5 points either way here, and it misses the weekly cycle in how people use the site. Marking it failed is the same mistake in the other direction, since the evidence the criterion asks for has not come in either way.

  11. 11In week 1 after launch, 41% of searches ended in a top-3 click. In week 2, after you added a photo to each result, it was 49% over 4,500 searches. A newsletter that week also brought many first-time visitors. What experiment would tell you whether the photos caused the rise?
    Show the answer

    Answer B. The interval says the rise is unlikely to be noise in 4,500 searches. It says nothing about the cause, because week 2 differed from week 1 in two ways: the photos and the newsletter's new visitors. Splitting visitors within the same weeks lets the newsletter, holidays and everything else reach both halves alike, so the difference between halves comes from the photos. Week 3 against week 1 still compares two different weeks, and a judge's opinion of the layout says nothing about what real visitors clicked.