What a Green Check Proves
An agent finishes a feature and reports: "All tests pass, type check is clean, CI is green, ready to merge." The pull request shows a green check. Three small hunks in the diff tell a different story:
// src/cart.test.ts
- expect(applyDiscount(cart, "SAVE10")).toEqual({ totalCents: 9000 });
+ expect(applyDiscount(cart, "SAVE10")).toBeDefined(); // tsconfig.json
- "strict": true,
+ "strict": false, # .github/workflows/ci.yml
integration:
+ if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latestEvery word of the agent's report can be true while the code is broken. A green check means something narrower than "the code works": every check that ran passed, on the commit CI checked out, in the pipeline's environment. Each part of that sentence is a place where a failure can be hidden.
What the pipeline does
Continuous integration, or CI, is a machine that is neither yours nor the agent's. On every push it checks out the code into an empty directory, installs dependencies from the lockfile, and runs a list of commands. Any command that exits with a non-zero status fails the run, and the result is attached to the pull request, where reviewers can see it.
For a push, the code is the pushed commit. For a pull request, GitHub's default is a temporary merge of the branch into the target branch, so the pipeline tests what the target would look like after the merge.
Starting from an empty directory matters because your laptop and the agent's sandbox carry state that the repository does not: untracked files, environment variables, globally installed tools, caches, a database someone migrated by hand. A check that passes on a clean machine passes mostly because of what is in the commit.
Here is a small pipeline in GitHub Actions syntax. Other CI systems use different syntax for the same parts.
name: ci
on: [push, pull_request]
jobs:
checks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- run: npm ci
- run: npx tsc --noEmit
- run: npx eslint .
- run: npm test
integration:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: test
ports: ["5432:5432"]
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- run: npm ci
- run: npm run test:integration
env:
DATABASE_URL: postgres://postgres:test@localhost:5432/postgresnpm ci installs exactly what the lockfile says and fails if the lockfile and package.json disagree. The .nvmrc file chooses the Node version, down to the patch only if it names one. So the lockfile pins your dependencies and .nvmrc pins your runtime. The health check under options makes the runner wait until PostgreSQL accepts connections before the integration steps start.
Other inputs still move. GitHub updates the ubuntu-latest runner image, postgres:16 floats across minor releases, and actions/checkout@v4 follows a tag its maintainers can move unless you pin the action to a commit SHA. A run can fail on an unchanged commit because one of those moved.

.nvmrc pin the dependencies and the runtime, but the runner image, the Postgres image and the checkout action can move. A run on an unchanged commit can fail because one of them did.A pipeline that only reports results still lets a red change merge. It becomes a merge gate when the repository refuses to merge a pull request until named checks pass. On GitHub these are required status checks in the branch protection settings. Those settings live in the repository configuration, outside any pull request, so a change cannot remove a check name from the required list. It can still edit the workflow file that produces that check, which changes what the check does.
If a required check never reports, the pull request waits. GitHub does report a job that an if: condition skipped as successful, so a required job with a condition can pass without running a single test.
Tests that cannot fail
A skipped test stays in the file and still reads like a requirement, but it checks nothing. Test frameworks all have a way to skip: test.skip and it.skip in JavaScript, @pytest.mark.skip in Python, t.Skip() in Go. Adding .only to one test is a quieter version, because it tells the runner to skip every other test in that file.
The test runner's summary line shows the damage if you look: "211 passed, 1 skipped" on a branch where main says "212 passed." A rising skip count is a signal worth tracking. A test that has to be disabled needs a written reason and a linked issue, because nothing else will remind anyone to turn it back on.
Skipping is the loud way to disable a test. A weakened assertion is the quiet way, because the test still runs and still passes.
In the first hunk of the opening diff, toEqual({ totalCents: 9000 }) became toBeDefined(). The original checks the discount arithmetic. The replacement passes for any return value except undefined, including a wrong total, so a change that stops applying the discount turns nothing red. Other weak forms are toBeTruthy(), a try block with an empty catch, and a snapshot regenerated with -u to match broken output.
Here both assertions run against a version of applyDiscount that never applies the code.
// Just enough of a test runner's `expect` to show the difference.
const expect = (actual: unknown) => ({
toEqual(expected: unknown) {
const [a, e] = [JSON.stringify(actual), JSON.stringify(expected)];
if (a !== e) throw new Error(`expected ${e}, got ${a}`);
},
toBeDefined() {
if (actual === undefined) throw new Error("expected a value, got undefined");
},
});
function test(name: string, body: () => void) {
try {
body();
console.log(`pass ${name}`);
} catch (e) {
console.log(`FAIL ${name}: ${(e as Error).message}`);
}
}
// A bug: the code is compared in lower case, so "SAVE10" never applies.
const applyDiscount = (cart: { totalCents: number }, code: string) =>
code === "save10" ? { totalCents: Math.round(cart.totalCents * 0.9) } : { ...cart };
const cart = { totalCents: 10000 };
test("old assertion", () => expect(applyDiscount(cart, "SAVE10")).toEqual({ totalCents: 9000 }));
test("new assertion", () => expect(applyDiscount(cart, "SAVE10")).toBeDefined());Output
In review, find every test whose state or assertions changed in the diff and ask which criterion it covered.
Conditional skips are harder to see. describe.skipIf(!process.env.PAYMENTS_KEY) runs the payment tests on a machine that has the key and silently skips them on one that does not. If CI lacks the key, the suite is green there and the payment tests have never run in CI. A test that needs an environment variable should fail loudly when it is missing.
Suppressed warnings
Checkers can be switched off for one line or for a whole project. A comment does the first: // eslint-disable-next-line, // @ts-ignore, # type: ignore and # noqa in Python, //nolint in Go. A setting does the second, such as "strict": false in the second hunk of the opening diff, which turns a family of checks off everywhere.
A line-sized suppression looks like this.
async function closeAccount(id: string) {
// eslint-disable-next-line @typescript-eslint/no-floating-promises
auditLog.write({ event: "closed", id });
return db.accounts.delete(id);
}The rule reports that auditLog.write returns a promise nobody awaits or handles. That is true, and it is a real hazard: if the write fails, nothing in this function sees it, and on current Node versions an unhandled rejection ends the process. With the comment, the lint run is clean and a test with a working log passes. The fix is to await the call or handle its failure.

closeAccount never awaits the audit write, so if the write fails nothing in the function sees it, and on current Node versions the unhandled rejection ends the process. The lint rule that reports this is silenced, so the lint run is clean.The project-wide switch hides more. With strictNullChecks off, which is part of strict, a function can call user.email.split("@") on a User whose email is optional, and the checker stays silent for every file at once. The same effect comes from narrowing include in tsconfig.json so the checker never reads some directories.
A suppression removes that checker's evidence about the line. The line still runs in tests, but no one is checking what the checker would have said about it. Search the diff for suppression comments and for changes to tsconfig.json, lint configuration and their equivalents, and for each one ask what the warning said and whether the warning was right. Usually the fix is to handle the case the checker found.
Linters can enforce this too: typescript-eslint's ban-ts-comment rule can reject @ts-ignore or require a written reason next to it. @ts-expect-error is the less dangerous form, because it becomes an error itself once the underlying problem is gone.
A pipeline that runs part of the suite
Green only covers the commands the pipeline runs, and a repository can easily hold more tests than CI executes. A test config with include: ["src/**/*.test.ts"] never finds the files in tests/integration/. A test:e2e script that someone runs by hand before releases never runs on a pull request. A matrix entry is excluded because it was flaky, and its platform is never tested again.
Conditions on a job do the same thing with less to see. The third hunk of the opening diff adds if: github.ref == 'refs/heads/main' to the integration job, so it runs only on main. On a pull request, and on each push to its branch, the job is skipped, and GitHub reports a skipped job as successful, so the check next to the merge button is green. The integration tests run after the merge, where a failure blocks nobody.
A paths: filter on the workflow behaves differently: when no changed file matches, the workflow never starts, and a required check that never reports leaves the pull request waiting.

paths: filter, the workflow never starts and a required check leaves the pull request waiting. If the job's if: condition is false, the job is skipped and reported as a success without running a test.With the narrow include, the hand-run script, the dropped matrix entry and the if: condition, every visible check is green and a broken change can merge. A required workflow with a paths: filter blocks the merge instead, though its tests still never ran. Either way, the question that exposes the gap is mechanical. List every test command and test directory in the repository, then find each one in the CI configuration, together with the conditions that decide when it runs. Anything you cannot find, or that does not run on pull requests, is unchecked on every merge. When a diff touches the workflow file or the test config, read that part first, because it changes what every future green means.
Green in the sandbox
An agent that runs the tests in its own environment and reports "all passing" is reporting on its environment. That environment may have a .env file with an API key, a fixture the agent generated that .gitignore keeps out of the commit, a database it set up by hand, or a different runtime version. The report is true there and says nothing about CI.
The block below runs one small suite in three places. One test is skipped when PAYMENTS_KEY is missing, like the describe.skipIf above, and one reads a fixture the agent generated and .gitignore kept out of the commit.
type Env = { name: string; vars: Record<string, string>; files: string[] };
type Test = { name: string; skipIf?: (env: Env) => boolean; run: (env: Env) => void };
const tests: Test[] = [
{ name: "cart totals", run: () => {} },
// In the real suite: describe.skipIf on a missing PAYMENTS_KEY
{ name: "charges the card", skipIf: (env) => !env.vars.PAYMENTS_KEY, run: () => {} },
{
name: "reads the price fixture",
run: (env) => {
if (!env.files.includes("fixtures/prices.json")) throw new Error("ENOENT: fixtures/prices.json");
},
},
];
function runSuite(env: Env) {
let passed = 0, failed = 0, skipped = 0;
const failures: string[] = [];
for (const t of tests) {
if (t.skipIf?.(env)) { skipped++; continue; }
try { t.run(env); passed++; } catch (e) { failed++; failures.push(` FAIL ${t.name}: ${(e as Error).message}`); }
}
const exitCode = failed > 0 ? 1 : 0;
console.log(`${env.name.padEnd(24)} ${passed} passed, ${failed} failed, ${skipped} skipped -> exit ${exitCode}`);
for (const f of failures) console.log(f);
}
// The agent's sandbox has a key in .env and a fixture it generated; .gitignore keeps both out of the commit.
runSuite({ name: "agent sandbox", vars: { PAYMENTS_KEY: "sk_test_123" }, files: ["fixtures/prices.json"] });
runSuite({ name: "CI", vars: {}, files: [] });
runSuite({ name: "CI, fixture committed", vars: {}, files: ["fixtures/prices.json"] });Output
The sandbox reports three passes. CI fails, which is the useful outcome, because the missing fixture throws. Once someone commits the fixture, CI exits 0 with "1 skipped" in the summary, and the payment test has still never run there.
The green that counts is the one CI attaches to the commit. Ask for the CI run, never the agent's summary of a local run. When a test passes locally and fails in CI, git status --ignored lists the files that exist on your machine but not in the repository, and a fresh clone into an empty directory reproduces CI's starting point.
Fast enough to run everything
A slow suite creates pressure to run less of it. People stop running it before pushing, and a twenty-minute job is the first one someone proposes to cut.
Take a suite that runs in 20 minutes on one machine: 1 minute of type checking, 1 of linting, 2 of unit tests and 16 of integration tests. Split into separate jobs that run in parallel, the wall-clock time drops to the longest job, about 16 minutes plus setup. Then shard the integration tests across four machines; Vitest, Jest and Playwright all accept a --shard=1/4 style flag. Each shard takes about 4 minutes, and with a minute of setup per job the whole pipeline finishes in about 5 minutes with every test still running.

Test selection cuts time further by running only the tests a change can affect. Locally that is what watch modes and commands like vitest related do, and it gives feedback in seconds while you work. For the merge gate, selection is safe only when the tool knows the real dependency graph, as build systems like Bazel do. Otherwise the gate should run everything.
What green cannot prove
A green pipeline says the checks that ran passed on the code CI checked out, in that environment. It says nothing about behavior no test covers, tests that were skipped or cannot fail, or production conditions the pipeline does not reproduce, such as real traffic, real third-party services and production configuration. Code under an ignore comment still runs in tests, but the type or lint check it escaped has said nothing about it. Whether the tests assert anything meaningful is the subject of tests as executable specifications.
A diff that changes tests, suppression comments, compiler settings or pipeline configuration changes what green means for every later commit, so read those lines before the code. A CODEOWNERS file, together with the branch protection setting that requires review from code owners, can make changes to .github/workflows/, the test configuration and tsconfig.json need approval from the people who own the pipeline.
Checking the running system against its criteria
CI checks a commit on a clean machine, once. The deployed system keeps changing after that: someone edits its configuration, its data grows, a third-party API changes its answers, a certificate expires. A criterion that passed in CI can be false in production a week later without a single new commit.
So the criteria a user would feel also run as probes against the live system, on a schedule. A probe calls the deployed app the way a user would and compares what comes back with the criterion. It runs every few minutes, and a failure alerts someone the way a failing test blocks a merge.
// Runs every 15 minutes against production, as a dedicated probe account.
// Criterion: SAVE10 takes 10% off a cart totalling 10000 cents.
// probe-item-100 is a hidden product priced at 10000 cents on the server.
const res = await fetch("https://shop.example.com/api/cart/quote", {
method: "POST",
headers: { "content-type": "application/json", authorization: `Bearer ${process.env.PROBE_TOKEN}` },
body: JSON.stringify({ items: [{ sku: "probe-item-100", qty: 1 }], code: "SAVE10" }),
});
const quote = res.ok ? await res.json() : null;
if (quote?.totalCents !== 9000) {
throw new Error(`criterion "SAVE10 takes 10% off" failed in production: status ${res.status}, total ${quote?.totalCents}`);
}The probe asks for a quote, so it never charges a card or touches a real customer's cart. Choose criteria that a read-only request or a probe account can check: a discount quote, a page that must load, a request for another account's order that must return 404. Name each probe after its criterion, so a red probe says which promise the live system broke, and keep the history of results. A criterion that fails at 03:00 every night points at whatever runs at 03:00, such as a backup or a batch import.
A green pipeline describes a commit at the moment it merged. A probe against the running system tells you whether its criteria still hold today.