Section 2 of 9
Verification and Evals
Proving You Got What You Wanted
Your criteria say what you want; tests and evals check them on every change, and the loop climbs the build toward them.
Why it matters when agents do the typing
Intent lives in the written criteria of the ideal state, and tests and evals are their executable form: each criterion carries the cheapest check that could prove it false, from an exact assert to a calibrated judge to a number from production. A check gives the same verdict on every run and does not care who is asking, so it keeps measuring long after the conversation that produced the code is gone, and the code can be thrown away and rebuilt against the same checks. When the same process writes both the code and its checks, both can carry the same misunderstanding, so the engineer owns what the checks assert, scores every build against them, and sharpens the criteria whenever a build passes them and is still wrong.
The course: 6 lessons, about 55 minutes of reading, in the order below.
-
2.1
Tests as Executable Specifications
Core 3 · Depth: do
Test-first thinking without the ritual: every behavior has a test, each test comes from a requirement, and nothing is done until the tests say so.
A mistake it teaches you to catch: A failing test 'fixed' by changing its expected value to match the buggy output.
Lesson: Where the Expected Value Comes From 9 min
-
2.2
Engineering Robust Tests
Core 4 · Depth: do
Choosing the right level of test, controlling time and randomness, and building suites that stay fast, deterministic and trustworthy.
A mistake it teaches you to catch: Everything mocked, so the tests never touch a real database and miss every query bug.
Lesson: A Test Should Give the Same Answer Every Time 8 min
-
2.3
Property-Based Testing and Test Quality
Depth: do
Checking invariants across thousands of generated inputs, and measuring whether your tests would actually catch a bug.
A mistake it teaches you to catch: Code that passes every hand-written example and fails on an input nobody thought of, such as an empty string, a huge number or combining characters.
Lesson: Let the Computer Choose the Inputs 8 min
-
2.4
Continuous Testing and Fast Feedback
Core 5 · Depth: do
Running every check on every change so you know within minutes, and ideally seconds, when something breaks.
A mistake it teaches you to catch: A failing test marked as skipped so the pipeline turns green.
Lesson: What a Green Check Proves 10 min
-
2.5
Evals: Measuring Any Output Against Its Criteria
Core 6 · Depth: do
Measuring any output an agent produces, from a model's reply to a report or a page, against its written criteria with the cheapest check that works, so a change is judged on evidence.
A mistake it teaches you to catch: A change shipped because its output looked better on three hand-picked examples.
Lesson: Evidence That an Output Is Good 10 min
-
2.6
The Loop: Build, Evaluate, Iterate
Core 7 · Depth: do
Scoring a build against its written criteria, fixing what failed, scoring everything again, and sharpening the criteria themselves, one change at a time, until the build is what you wanted.
A mistake it teaches you to catch: An agent changes several things in one round, the total score rises, and nobody can say which change helped and which one broke a criterion that used to pass.
Lesson: Score It, Change One Thing, Score It Again 10 min
Section test: Verification and Evals
11 questions across the 6 lessons above. Take it after the course; each answer explains itself.