Building Software
Sections

Contents Section 2, Verification and Evals

Continuous Testing and Fast Feedback

Running every check on every change so you know within minutes, and ideally seconds, when something breaks.

Section 2 of 9, subsection 4 of 6. Depth: do. Core 5 of 25. A 10-minute read.

Mistakes to catch in review

  1. A failing test marked as skipped so the pipeline turns green.

  2. Type-checker or linter warnings suppressed with ignore comments instead of fixed.

  3. A pipeline that runs only part of the suite, letting a broken integration merge.

  4. An agent that reports the suite green from its own sandbox, where a fixture file or an environment variable exists that CI does not have.

What a Green Check Proves

An agent finishes a feature and reports: "All tests pass, type check is clean, CI is green, ready to merge." The pull request shows a green check. Three small hunks in the diff tell a different story:

Diff
  // src/cart.test.ts
- expect(applyDiscount(cart, "SAVE10")).toEqual({ totalCents: 9000 });
+ expect(applyDiscount(cart, "SAVE10")).toBeDefined();
Diff
  // tsconfig.json
-   "strict": true,
+   "strict": false,
Diff
  # .github/workflows/ci.yml
    integration:
+     if: github.ref == 'refs/heads/main'
      runs-on: ubuntu-latest

Every word of the agent's report can be true while the code is broken. A green check means something narrower than "the code works": every check that ran passed, on the commit CI checked out, in the pipeline's environment. Each part of that sentence is a place where a failure can be hidden.

What the pipeline does

Continuous integration, or CI, is a machine that is neither yours nor the agent's. On every push it checks out the code into an empty directory, installs dependencies from the lockfile, and runs a list of commands. Any command that exits with a non-zero status fails the run, and the result is attached to the pull request, where reviewers can see it.

For a push, the code is the pushed commit. For a pull request, GitHub's default is a temporary merge of the branch into the target branch, so the pipeline tests what the target would look like after the merge.

Starting from an empty directory matters because your laptop and the agent's sandbox carry state that the repository does not: untracked files, environment variables, globally installed tools, caches, a database someone migrated by hand. A check that passes on a clean machine passes mostly because of what is in the commit.

Here is a small pipeline in GitHub Actions syntax. Other CI systems use different syntax for the same parts.

YAML
name: ci
on: [push, pull_request]

jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version-file: .nvmrc
      - run: npm ci
      - run: npx tsc --noEmit
      - run: npx eslint .
      - run: npm test

  integration:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version-file: .nvmrc
      - run: npm ci
      - run: npm run test:integration
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/postgres

npm ci installs exactly what the lockfile says and fails if the lockfile and package.json disagree. The .nvmrc file chooses the Node version, down to the patch only if it names one. So the lockfile pins your dependencies and .nvmrc pins your runtime. The health check under options makes the runner wait until PostgreSQL accepts connections before the integration steps start.

Other inputs still move. GitHub updates the ubuntu-latest runner image, postgres:16 floats across minor releases, and actions/checkout@v4 follows a tag its maintainers can move unless you pin the action to a commit SHA. A run can fail on an unchanged commit because one of those moved.

A pill, 'Same commit', points down into a purple box, 'CI run: result can change'. On the left, under 'Pinned', solid arrows come from 'Lockfile: pins dependencies' and '.nvmrc: pins the runtime'. On the right, under 'Moves', dashed arrows come from 'ubuntu-latest: GitHub updates the image', 'postgres:16: floats across minor releases' and 'actions/checkout@v4: a tag maintainers can move'. A note reads: same commit, different run.
The lockfile and .nvmrc pin the dependencies and the runtime, but the runner image, the Postgres image and the checkout action can move. A run on an unchanged commit can fail because one of them did.

A pipeline that only reports results still lets a red change merge. It becomes a merge gate when the repository refuses to merge a pull request until named checks pass. On GitHub these are required status checks in the branch protection settings. Those settings live in the repository configuration, outside any pull request, so a change cannot remove a check name from the required list. It can still edit the workflow file that produces that check, which changes what the check does.

If a required check never reports, the pull request waits. GitHub does report a job that an if: condition skipped as successful, so a required job with a condition can pass without running a single test.

Tests that cannot fail

A skipped test stays in the file and still reads like a requirement, but it checks nothing. Test frameworks all have a way to skip: test.skip and it.skip in JavaScript, @pytest.mark.skip in Python, t.Skip() in Go. Adding .only to one test is a quieter version, because it tells the runner to skip every other test in that file.

The test runner's summary line shows the damage if you look: "211 passed, 1 skipped" on a branch where main says "212 passed." A rising skip count is a signal worth tracking. A test that has to be disabled needs a written reason and a linked issue, because nothing else will remind anyone to turn it back on.

Skipping is the loud way to disable a test. A weakened assertion is the quiet way, because the test still runs and still passes.

In the first hunk of the opening diff, toEqual({ totalCents: 9000 }) became toBeDefined(). The original checks the discount arithmetic. The replacement passes for any return value except undefined, including a wrong total, so a change that stops applying the discount turns nothing red. Other weak forms are toBeTruthy(), a try block with an empty catch, and a snapshot regenerated with -u to match broken output.

Here both assertions run against a version of applyDiscount that never applies the code.

TypeScriptRuns in your browser
// Just enough of a test runner's `expect` to show the difference.
const expect = (actual: unknown) => ({
  toEqual(expected: unknown) {
    const [a, e] = [JSON.stringify(actual), JSON.stringify(expected)];
    if (a !== e) throw new Error(`expected ${e}, got ${a}`);
  },
  toBeDefined() {
    if (actual === undefined) throw new Error("expected a value, got undefined");
  },
});

function test(name: string, body: () => void) {
  try {
    body();
    console.log(`pass  ${name}`);
  } catch (e) {
    console.log(`FAIL  ${name}: ${(e as Error).message}`);
  }
}

// A bug: the code is compared in lower case, so "SAVE10" never applies.
const applyDiscount = (cart: { totalCents: number }, code: string) =>
  code === "save10" ? { totalCents: Math.round(cart.totalCents * 0.9) } : { ...cart };

const cart = { totalCents: 10000 };
test("old assertion", () => expect(applyDiscount(cart, "SAVE10")).toEqual({ totalCents: 9000 }));
test("new assertion", () => expect(applyDiscount(cart, "SAVE10")).toBeDefined());

In review, find every test whose state or assertions changed in the diff and ask which criterion it covered.

Conditional skips are harder to see. describe.skipIf(!process.env.PAYMENTS_KEY) runs the payment tests on a machine that has the key and silently skips them on one that does not. If CI lacks the key, the suite is green there and the payment tests have never run in CI. A test that needs an environment variable should fail loudly when it is missing.

Suppressed warnings

Checkers can be switched off for one line or for a whole project. A comment does the first: // eslint-disable-next-line, // @ts-ignore, # type: ignore and # noqa in Python, //nolint in Go. A setting does the second, such as "strict": false in the second hunk of the opening diff, which turns a family of checks off everywhere.

A line-sized suppression looks like this.

TypeScript
async function closeAccount(id: string) {
  // eslint-disable-next-line @typescript-eslint/no-floating-promises
  auditLog.write({ event: "closed", id });
  return db.accounts.delete(id);
}

The rule reports that auditLog.write returns a promise nobody awaits or handles. That is true, and it is a real hazard: if the write fails, nothing in this function sees it, and on current Node versions an unhandled rejection ends the process. With the comment, the lint run is clean and a test with a working log passes. The fix is to await the call or handle its failure.

A pill, closeAccount(id), has arrows to auditLog.write, 'promise nobody awaits', and to db.accounts.delete, 'returned'. Between them, no-floating-promises, 'silenced by a comment', has a dashed arrow up to auditLog.write. auditLog.write leads to 'Write fails: nothing in the function sees it', which leads down to a purple 'Process ends: unhandled rejection, current Node'. A note reads: lint clean, hazard still there.
closeAccount never awaits the audit write, so if the write fails nothing in the function sees it, and on current Node versions the unhandled rejection ends the process. The lint rule that reports this is silenced, so the lint run is clean.

The project-wide switch hides more. With strictNullChecks off, which is part of strict, a function can call user.email.split("@") on a User whose email is optional, and the checker stays silent for every file at once. The same effect comes from narrowing include in tsconfig.json so the checker never reads some directories.

A suppression removes that checker's evidence about the line. The line still runs in tests, but no one is checking what the checker would have said about it. Search the diff for suppression comments and for changes to tsconfig.json, lint configuration and their equivalents, and for each one ask what the warning said and whether the warning was right. Usually the fix is to handle the case the checker found.

Linters can enforce this too: typescript-eslint's ban-ts-comment rule can reject @ts-ignore or require a written reason next to it. @ts-expect-error is the less dangerous form, because it becomes an error itself once the underlying problem is gone.

A pipeline that runs part of the suite

Green only covers the commands the pipeline runs, and a repository can easily hold more tests than CI executes. A test config with include: ["src/**/*.test.ts"] never finds the files in tests/integration/. A test:e2e script that someone runs by hand before releases never runs on a pull request. A matrix entry is excluded because it was flaky, and its platform is never tested again.

Conditions on a job do the same thing with less to see. The third hunk of the opening diff adds if: github.ref == 'refs/heads/main' to the integration job, so it runs only on main. On a pull request, and on each push to its branch, the job is skipped, and GitHub reports a skipped job as successful, so the check next to the merge button is green. The integration tests run after the merge, where a failure blocks nobody.

A paths: filter on the workflow behaves differently: when no changed file matches, the workflow never starts, and a required check that never reports leaves the pull request waiting.

A flowchart. 'Pull request' leads to a diamond, 'paths: match?'. 'No' leads down to 'Never starts: required check waits'. 'Yes' leads to a second diamond, 'Job if: true?'. 'False' leads down to a purple 'Skipped: reported as success'; 'true' leads to 'Job runs: tests pass or fail'. A note beside the skipped box reads: green, no tests ran.
If no changed file matches the paths: filter, the workflow never starts and a required check leaves the pull request waiting. If the job's if: condition is false, the job is skipped and reported as a success without running a test.

With the narrow include, the hand-run script, the dropped matrix entry and the if: condition, every visible check is green and a broken change can merge. A required workflow with a paths: filter blocks the merge instead, though its tests still never ran. Either way, the question that exposes the gap is mechanical. List every test command and test directory in the repository, then find each one in the CI configuration, together with the conditions that decide when it runs. Anything you cannot find, or that does not run on pull requests, is unchecked on every merge. When a diff touches the workflow file or the test config, read that part first, because it changes what every future green means.

Green in the sandbox

An agent that runs the tests in its own environment and reports "all passing" is reporting on its environment. That environment may have a .env file with an API key, a fixture the agent generated that .gitignore keeps out of the commit, a database it set up by hand, or a different runtime version. The report is true there and says nothing about CI.

The block below runs one small suite in three places. One test is skipped when PAYMENTS_KEY is missing, like the describe.skipIf above, and one reads a fixture the agent generated and .gitignore kept out of the commit.

TypeScriptRuns in your browser
type Env = { name: string; vars: Record<string, string>; files: string[] };
type Test = { name: string; skipIf?: (env: Env) => boolean; run: (env: Env) => void };

const tests: Test[] = [
  { name: "cart totals", run: () => {} },
  // In the real suite: describe.skipIf on a missing PAYMENTS_KEY
  { name: "charges the card", skipIf: (env) => !env.vars.PAYMENTS_KEY, run: () => {} },
  {
    name: "reads the price fixture",
    run: (env) => {
      if (!env.files.includes("fixtures/prices.json")) throw new Error("ENOENT: fixtures/prices.json");
    },
  },
];

function runSuite(env: Env) {
  let passed = 0, failed = 0, skipped = 0;
  const failures: string[] = [];
  for (const t of tests) {
    if (t.skipIf?.(env)) { skipped++; continue; }
    try { t.run(env); passed++; } catch (e) { failed++; failures.push(`  FAIL ${t.name}: ${(e as Error).message}`); }
  }
  const exitCode = failed > 0 ? 1 : 0;
  console.log(`${env.name.padEnd(24)} ${passed} passed, ${failed} failed, ${skipped} skipped -> exit ${exitCode}`);
  for (const f of failures) console.log(f);
}

// The agent's sandbox has a key in .env and a fixture it generated; .gitignore keeps both out of the commit.
runSuite({ name: "agent sandbox", vars: { PAYMENTS_KEY: "sk_test_123" }, files: ["fixtures/prices.json"] });
runSuite({ name: "CI", vars: {}, files: [] });
runSuite({ name: "CI, fixture committed", vars: {}, files: ["fixtures/prices.json"] });

The sandbox reports three passes. CI fails, which is the useful outcome, because the missing fixture throws. Once someone commits the fixture, CI exits 0 with "1 skipped" in the summary, and the payment test has still never run there.

The green that counts is the one CI attaches to the commit. Ask for the CI run, never the agent's summary of a local run. When a test passes locally and fails in CI, git status --ignored lists the files that exist on your machine but not in the repository, and a fresh clone into an empty directory reproduces CI's starting point.

Fast enough to run everything

A slow suite creates pressure to run less of it. People stop running it before pushing, and a twenty-minute job is the first one someone proposes to cut.

Take a suite that runs in 20 minutes on one machine: 1 minute of type checking, 1 of linting, 2 of unit tests and 16 of integration tests. Split into separate jobs that run in parallel, the wall-clock time drops to the longest job, about 16 minutes plus setup. Then shard the integration tests across four machines; Vitest, Jest and Playwright all accept a --shard=1/4 style flag. Each shard takes about 4 minutes, and with a minute of setup per job the whole pipeline finishes in about 5 minutes with every test still running.

Three rows. One machine: 20 min, 1 + 1 + 2 + 16 minutes, in turn. Split into jobs, parallel jobs: about 16 min plus setup, the longest job being integration. Shard integration across 4: four shards side by side, each about 4 min, all feeding one highlighted total, about 5 min, 4 + 1 setup. A note reads: every test still runs.
As parallel jobs, the 16-minute integration job sets the pace. Sharding it across four machines brings the whole pipeline to about 5 minutes with every test still running.

Test selection cuts time further by running only the tests a change can affect. Locally that is what watch modes and commands like vitest related do, and it gives feedback in seconds while you work. For the merge gate, selection is safe only when the tool knows the real dependency graph, as build systems like Bazel do. Otherwise the gate should run everything.

What green cannot prove

A green pipeline says the checks that ran passed on the code CI checked out, in that environment. It says nothing about behavior no test covers, tests that were skipped or cannot fail, or production conditions the pipeline does not reproduce, such as real traffic, real third-party services and production configuration. Code under an ignore comment still runs in tests, but the type or lint check it escaped has said nothing about it. Whether the tests assert anything meaningful is the subject of tests as executable specifications.

A diff that changes tests, suppression comments, compiler settings or pipeline configuration changes what green means for every later commit, so read those lines before the code. A CODEOWNERS file, together with the branch protection setting that requires review from code owners, can make changes to .github/workflows/, the test configuration and tsconfig.json need approval from the people who own the pipeline.

Checking the running system against its criteria

CI checks a commit on a clean machine, once. The deployed system keeps changing after that: someone edits its configuration, its data grows, a third-party API changes its answers, a certificate expires. A criterion that passed in CI can be false in production a week later without a single new commit.

So the criteria a user would feel also run as probes against the live system, on a schedule. A probe calls the deployed app the way a user would and compares what comes back with the criterion. It runs every few minutes, and a failure alerts someone the way a failing test blocks a merge.

TypeScript
// Runs every 15 minutes against production, as a dedicated probe account.
// Criterion: SAVE10 takes 10% off a cart totalling 10000 cents.
// probe-item-100 is a hidden product priced at 10000 cents on the server.
const res = await fetch("https://shop.example.com/api/cart/quote", {
  method: "POST",
  headers: { "content-type": "application/json", authorization: `Bearer ${process.env.PROBE_TOKEN}` },
  body: JSON.stringify({ items: [{ sku: "probe-item-100", qty: 1 }], code: "SAVE10" }),
});
const quote = res.ok ? await res.json() : null;
if (quote?.totalCents !== 9000) {
  throw new Error(`criterion "SAVE10 takes 10% off" failed in production: status ${res.status}, total ${quote?.totalCents}`);
}

The probe asks for a quote, so it never charges a card or touches a real customer's cart. Choose criteria that a read-only request or a probe account can check: a discount quote, a page that must load, a request for another account's order that must return 404. Name each probe after its criterion, so a red probe says which promise the live system broke, and keep the history of results. A criterion that fails at 03:00 every night points at whatever runs at 03:00, such as a backup or a batch import.

A green pipeline describes a commit at the moment it merged. A probe against the running system tells you whether its criteria still hold today.

Check your understanding

7 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1In this workflow, web-tests is a required status check. A pull request changes only files under api/. What does the merge box show?
    YAML
    on:
      pull_request:
        paths: ["web/**"]
    
    jobs:
      web-tests:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - run: npm ci
          - run: npm test --workspace web
    Show the answer

    Answer B. No changed file matches web/**, so the workflow never starts and the required check never reports. GitHub keeps waiting for it. The skipped-counts-as-success rule is the tempting answer, but it applies to a job skipped by an if: condition inside a workflow that did start.

  2. 2CI runs npx vitest run with this config. The repository holds src/cart.test.ts, src/cart.spec.ts and tests/integration/checkout.test.ts. Which files does a green run cover?
    TypeScript
    import { defineConfig } from "vitest/config";
    
    export default defineConfig({
      test: { include: ["src/**/*.test.ts"] },
    });
    Show the answer

    Answer D. Setting include replaces Vitest's default pattern, so only files under src/ ending in .test.ts are collected. The default would have found .spec.ts files too, which is why the first option is tempting. The integration file sits outside src/ and never runs, while every check stays green.

  3. 3A month after this line was written, someone changes getId to return a number. What does the next tsc --noEmit in CI do?
    TypeScript
    // @ts-expect-error legacy client still returns ids as strings
    const id: number = legacyClient.getId();
    Show the answer

    Answer A. @ts-expect-error asserts that the next line has a type error, and TypeScript reports an error when it does not. That makes the suppression expire once the problem it hid is gone. @ts-ignore is the one that stays silent forever, which is why linters can ban it or require a written reason.

  4. 4A single CI job takes 30 minutes: 2 of type checking, 1 of linting, 3 of unit tests and 24 of integration tests. You split these into four parallel jobs and shard the integration tests across 6 machines. Each job spends 1 minute on setup. About how long until the pull request shows a result?
    Show the answer

    Answer C. Parallel jobs finish when the longest one does. Each integration shard runs 24 / 6 = 4 minutes plus 1 of setup, which is 5, and the unit job takes 3 + 1 = 4. The same tests do still run, so the machine-minutes go up slightly, but you wait for the longest job. 25 minutes is what you get from splitting without sharding.

  5. 5To cut a 12-minute merge gate, an agent proposes running only the test files that the pull request itself changed. The repository has no build tool that tracks dependencies. Which broken change can now merge with a green check?
    Show the answer

    Answer D. checkout.test.ts did not change, so it is not selected, even though the code it tests did. Every other option involves a test file that changed, and those still run. Selection is safe at the gate only when a tool knows the real dependency graph; without one the gate runs everything.

  6. 6An agent reports "all 48 tests pass". CI fails one of them with ENOENT: fixtures/users.json, and the same test passes on the agent's machine. What do you check first?
    Show the answer

    Answer A. ENOENT means the path could not be found when the test opened it. Since the same test passes on the agent's machine, a fixture the agent generated and .gitignore kept out of the commit is the likeliest cause, and git status --ignored checks it in seconds. If the fixture is committed, look next at the working directory and at what creates or deletes the file during the run. Rerunning is tempting, but a pass on the rerun would not say why the file was missing.

  7. 7An agent made this change in a pull request titled "deflake auth test". Which regression can now pass?
    Diff
      const res = await request(app).get("/admin/reports").set("Cookie", viewerCookie);
    - expect(res.status).toBe(403);
    + expect([200, 403]).toContain(res.status);
    Show the answer

    Answer C. A 200 for a viewer is exactly the failure this test existed to catch, and the new assertion accepts it. A 500 is the tempting answer, but 500 is not in [200, 403], so that still fails. The test still runs and still checks the status, which is what makes a weakened assertion harder to spot than a skip.

You understand it when you can

Drill

An agent's pull request turns a red pipeline green by marking two tests as skipped, adding an ignore comment above a type error, and deleting the integration-test job from the CI configuration. Find each change that hides a failure, and state what the pipeline no longer proves.

Topics

Continuous Integration Pipelines
Automatically building and testing every change, with results attached to the change itself.
Types, Linters and Static Analysis
Checks that run without executing the code, including secret and dependency scanners, and catch whole classes of mistakes cheaply.
Merge Gates
Blocking changes that fail their checks, and treating a broken main branch as an emergency.
Fast Feedback
Keeping the suite fast enough to run on every change through parallelism, test selection and watch modes.
Reproducible Environments
Pinned toolchains, containers and lockfiles so local and CI results agree.
Verifying the Running System Against Its Criteria
Running criteria a user would feel as scheduled probes against the deployed system, named after the criterion they check, because configuration, data and outside services change after the commit was tested.

Further reading

Start here

  • Top 10 Rules For Continuous Integration. Dave Farley, 2020. 18-minute explainer.

    States the working rules behind CI, including commit often, never leave the build broken, fix or revert a red main immediately and keep the build fast, which are exactly the rules skipped tests and deleted jobs break.

Short reads

Watch

Books

  • Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Jez Humble and David Farley, 2010.

    Introduced the deployment pipeline: every commit runs through staged automated checks, and a failure at any stage stops the change from going further.

  • Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. Nicole Forsgren, Jez Humble and Gene Kim, 2018.

    Presents the survey research linking test automation, trunk-based development and fast feedback to delivery performance, which is the evidence behind treating a slow or skipped suite as a real cost.

  • Continuous Delivery Pipelines: How to Build Better Software Faster. Dave Farley, 2021.

    A short practical handbook on pipeline stages (commit tests, acceptance tests, version control of everything, infrastructure as code) and how to keep each stage fast and trusted.

Primary sources

  • Continuous Integration (Martin Fowler). Reference.

    The canonical, recently revised description of CI practices: a self-testing build on every push, fixing broken builds immediately, keeping builds fast and testing in a clone of production.

  • About protected branches (GitHub Docs). Manual.

    The official reference for required status checks and other branch rules that stop a pull request from merging until named CI jobs pass, which is the mechanical form of a merge gate.