Building Software
Sections

Contents Section 7, Operations

Debugging

Finding the cause of a failure by reproducing it and running experiments, instead of guessing at fixes.

Section 7 of 9, subsection 1 of 5. Depth: do. Core 16 of 25. A 9-minute read.

Mistakes to catch in review

  1. A symptom patched with a null check, a retry or a broad try/catch while the actual cause stays in place.

  2. A fix declared successful without the original failure ever being reproduced.

  3. The same error pasted back to an agent again and again while nobody forms a hypothesis about the cause.

Make It Fail, Then Ask Why

An agent hands you a two-line diff. A nightly job that builds a daily sales report had been crashing about once a month with TypeError: Cannot read properties of undefined (reading 'total'). The agent changed one line so the lookup reads totalsByDay[key]?.total ?? 0, ran the tests, and reported that the crash is fixed. The tests pass and the crash reports stop.

You now have a report that never crashes, and you have no idea whether its numbers are right. The crash was the only signal that something upstream produced a bad value. The diff removed the signal and left the bad value where it was. Debugging is the work of finding where that value came from, and it has a method that works far better than guessing.

What a bug actually is

Every failure you see is the end of a chain. Somewhere, code produced a wrong value: a wrong key, a stale timestamp, a count off by one. That wrong value then travelled through the program, passed between functions, stored in variables, until some line tried to use it in a way that could not work. That last line is where the crash happens and where the stack trace points.

So there are two places in every bug. The defect is where the wrong value was created. The failure is where you noticed it. They can be one line apart or in different services. A stack trace tells you exactly where the failure is. It shows the defect only if the defective code is still on the call stack when the crash happens, and often it has already returned.

A left-to-right chain of four boxes. The first, highlighted, is the defect, where the wrong key is created. An arrow leads from it to passed along between functions, then to stored in variables, then to the failure, reading .total of undefined. Above the failure, a stack trace points down at it and only at it. Below the failure, a box holding the patch ?.total ?? 0 points up at the failure line it guards. A note under the defect reads: defect still upstream.
The wrong key is created upstream and only fails several steps later, which is the one place the stack trace points. The ?.total ?? 0 patch guards that failure line, so the defect keeps producing wrong keys with nothing left to complain.

This is why a null check at the crash site is the wrong fix here. The lookup must find the day, so a missing entry means something upstream went wrong, and defaulting to zero hides that. A null check is the right fix when absence is a valid state, such as a day that really had no sales. The ?.total ?? 0 change guards the failure line. The defect is still upstream, still producing wrong keys, and now nothing complains. The same logic applies to wrapping a handler in try/catch, or adding retries. A retry helps when the failure depends on something that changes between attempts, like a dropped network connection. When the failure comes from the code or its input, a retry runs the same defect again. If it fails every time, the retry only delays the error. If it fails some of the time, the retry hides it, and the defect keeps running on every attempt.

The question that exposes a symptom patch is: in the case that used to crash, what does the program produce now, and is that value correct? For the sales report, the answer is "zero dollars for that day", which is false.

Reproduce first

Before changing any code, try to make the failure occur on demand, and make that way as small as you can. When you can't reproduce it, test your hypotheses against whatever evidence was captured, such as logs, memory dumps and traces, and say plainly what remains uncertain.

The production job crashed "about once a month", on no particular weekday, which sounds random. The crash reports carried the job's input, and when you line them up they have one thing in common: every crash was a report for the 1st of a month. Here is the code involved.

TypeScript
// src/sales.ts
export function dayKey(isoDate: string): string {
  const d = new Date(isoDate);
  const mm = String(d.getMonth() + 1).padStart(2, "0");
  const dd = String(d.getDate()).padStart(2, "0");
  return `${d.getFullYear()}-${mm}-${dd}`;
}

export function dailyTotal(totalsByDay: Record<string, { total: number }>, isoDate: string): number {
  return totalsByDay[dayKey(isoDate)].total;
}

totalsByDay holds one entry per day of the current month, keyed like "2026-03-01". A reproduction starts from the real input and cuts away everything that turns out to be irrelevant: the database, the scheduler, the other days of the month. Each time you remove something, run it again. If it still fails, the thing you removed was not needed for the failure. If it stops failing, put it back, because it matters. What survives is a test like this one, which Node 22.18 and later run straight from the .ts file.

TypeScript
// test/daykey.regress.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { dayKey, dailyTotal } from "../src/sales.ts";

test("dayKey keeps the calendar date it was given", () => {
  assert.equal(dayKey("2026-03-01"), "2026-03-01");
});

test("the 1st of the month reads its own total", () => {
  const march = { "2026-03-01": { total: 412 } };
  assert.equal(dailyTotal(march, "2026-03-01"), 412);
});

On your laptop this test passes. In CI it passes too. It fails only when you run it the way production runs it:

Shell
TZ=UTC node --test test/daykey.regress.test.ts                   # passes
TZ=America/Los_Angeles node --test test/daykey.regress.test.ts   # fails

The production servers were configured for Pacific time. CI containers and many cloud machines default to UTC. That one environment variable separates a bug that never appears in testing from a test that fails every time.

A failing test is the best form a reproduction can take, because it doubles as the proof of the fix. The rule for accepting any fix, from an agent or a person, is to see this test fail before the change and pass after it. A fix accepted without that is a guess that happened to coincide with the crash reports going quiet, and an intermittent failure can go quiet for weeks with no change at all.

Debug by experiment

When an agent gets an error, the easy loop is to paste the error back and ask for another fix. Each round, the agent reads the same stack trace, makes a new guess about the cause, and changes some code. Nothing is learned between rounds, because the last guess, what it predicted and whether the prediction held were never written down. After five rounds you have five changes and no more understanding than you started with.

The alternative is to treat each step as an experiment. State a hypothesis about the cause. Say what you would observe if it were true. Then run the smallest check that could prove it false. Write all three down, including the ones that fail, because a disproved hypothesis narrows the search as much as a confirmed one.

Here is how that went for the sales report.

  • Hypothesis 1: the data loader skips the 1st of the month. Prediction: totalsByDay has no "2026-03-01" key. Check: print the keys. The key is there, with the right total. Disproved.
  • Hypothesis 2: the key being looked up is wrong. Prediction: dayKey("2026-03-01") returns something other than "2026-03-01". Check: call it under Pacific time. It returns "2026-02-28". Confirmed.
  • Hypothesis 3: other days are also shifted, and they only avoid crashing because the previous day exists in the map. Prediction: dayKey("2026-03-15") returns "2026-03-14" under Pacific time. Check: it does.

Hypothesis 3 is the important one. The crash on the 1st was the visible part of a larger bug. Every other day of the month, the report silently showed the previous day's total. The agent's ?? 0 patch would have hidden the only case that made noise and kept all the wrong numbers.

When you work with an agent, give it the experiment instead of the error. "Under TZ=America/Los_Angeles, what does dayKey('2026-03-01') return, and why?" is a question with a checkable answer. "It crashed again, here's the trace" invites another guess.

Why it happens

JavaScript's new Date("2026-03-01") parses the string as midnight UTC on that date. The ECMAScript spec treats date-only ISO strings as UTC, and date-time strings without an offset, like "2026-03-01T00:00", as local time. Midnight UTC on March 1 is 16:00 on February 28 in Los Angeles. getDate() and getMonth() read the date in the machine's local timezone, so they return the 28th of February. East of UTC, local time is ahead of UTC, the calendar date stays the same, and the bug never shows. That is why a developer in Berlin and a CI runner in UTC both saw green tests.

A flow from left to right. The string new Date("2026-03-01"), a date-only string, is parsed as UTC into one highlighted instant, midnight UTC on March 1. That instant fans out to three timezones. UTC reads March 1 and builds the key "2026-03-01", which is found. Europe/Berlin, ahead of UTC, also reads March 1 and builds "2026-03-01", found. Los Angeles reads 16:00 on February 28, so the local getters build the key "2026-02-28", which has no entry and crashes. A note reads: the 15th reads the 14th.
new Date("2026-03-01") is one instant, midnight UTC. In UTC and Berlin the local getters still read March 1, but in Los Angeles they read February 28, so the key misses on the 1st and quietly points at the wrong day on every other date.

A browser cannot change its own timezone, so the block below passes the zone in and reads the date the way getDate() would read it there. Near the end it runs the real dayKey in your machine's own zone, so that line depends on where you are.

TypeScriptRuns in your browser
const march: Record<string, { total: number }> = {
  "2026-03-01": { total: 412 },
  "2026-03-14": { total: 980 },
  "2026-03-15": { total: 655 },
};

function dayKeyIn(isoDate: string, timeZone: string): string {
  const parts = new Intl.DateTimeFormat("en-CA", { timeZone, year: "numeric", month: "2-digit", day: "2-digit" })
    .formatToParts(new Date(isoDate));
  const part = (type: string) => parts.find((p) => p.type === type)?.value;
  return `${part("year")}-${part("month")}-${part("day")}`;
}

console.log(`new Date("2026-03-01") is the instant ${new Date("2026-03-01").toISOString()}`);
for (const zone of ["UTC", "Europe/Berlin", "America/Los_Angeles"]) {
  for (const day of ["2026-03-01", "2026-03-15"]) {
    const key = dayKeyIn(day, zone);
    const total = march[key]?.total ?? "undefined, the crash";
    console.log(`${zone.padEnd(20)} ${day}: key ${key}, total ${total}`);
  }
}

function dayKey(isoDate: string): string {
  const d = new Date(isoDate);
  const mm = String(d.getMonth() + 1).padStart(2, "0");
  const dd = String(d.getDate()).padStart(2, "0");
  return `${d.getFullYear()}-${mm}-${dd}`;
}
const here = Intl.DateTimeFormat().resolvedOptions().timeZone;
console.log(`your machine (${here}): dayKey("2026-03-01") is ${dayKey("2026-03-01")}`);
console.log(`the fix, "2026-03-01".slice(0, 10), is ${"2026-03-01".slice(0, 10)} in every zone`);

In Los Angeles the 15th quietly reads the 14th's total, 980 instead of 655. That is the wrong number Hypothesis 3 predicted, and nothing in the output marks it as wrong.

Stack traces and logs each gave part of this. The trace said where: line 10 of sales.ts, reading .total of undefined. The job's log gave when and with what data: the input date, and the fact that every failure was on the 1st. The timezone came from comparing the environment where it failed with the environments where it did not. Read a trace for the location of the failure, then go to the logs and the environment for the data that led there.

A debugger is the other way to see the chain directly. Run the test with node --inspect-brk, set a breakpoint inside dayKey, and watch d and d.getDate() as they are computed. You see the wrong day appear on the line where it is created, which is the defect, instead of in the caller, where it is used.

Finding the commit with bisection

The report used to work. Somewhere between the last release and today, a commit introduced the bug, and knowing which one tells you what the author was trying to do and what else they changed.

Bisection finds it by halving. Mark one commit where the bug is present and one where it is absent. Test the commit halfway between. Whichever half contains the change from good to bad is where you look next. With 200 commits between the two, you need about 8 tests, since 2 to the 8th is 256. Git automates the whole search:

Shell
git bisect start
git bisect bad HEAD
git bisect good v1.4.0
git bisect run env TZ=America/Los_Angeles node --test test/daykey.regress.test.ts
git bisect reset

git bisect run checks out each candidate commit, runs your command, and reads its exit code: 0 means good, 125 means this commit cannot be tested and should be skipped, and most other non-zero codes mean bad. Two details matter. Keep the regression test untracked, in a new path that no candidate commit touches (or outside the checkout), because Git leaves untracked files in place as it checks out old commits, so the test exists at every step. And the command must fail only for the bug you are chasing. If a commit fails for another reason, such as the test being unable to import sales.ts there, the search follows the wrong trail. Have the command exit with 125 on those commits instead.

The block below runs the halving over 200 commits. The bug arrives in commit 137, and commits 95 to 104 had sales.ts moved to another folder, so the test cannot import it there. A careless test calls those commits bad; a careful one skips them.

TypeScriptRuns in your browser
type Verdict = "good" | "bad" | "skip";
const BUG_FROM = 137;
const moved = (c: number) => c >= 95 && c <= 104;
const careless = (c: number): Verdict => (moved(c) || c >= BUG_FROM ? "bad" : "good");
const careful = (c: number): Verdict => (moved(c) ? "skip" : c >= BUG_FROM ? "bad" : "good");

function bisect(name: string, test: (c: number) => Verdict) {
  let good = 0;
  let bad = 200;
  const seen: string[] = [];
  while (bad - good > 1) {
    let c = Math.floor((good + bad) / 2);
    // Git picks another commit nearby when one is skipped; this takes the next one up.
    while (test(c) === "skip" && c + 1 < bad) {
      seen.push(`${c} skip`);
      c++;
    }
    const v = test(c);
    if (v === "skip") break;
    seen.push(`${c} ${v}`);
    if (v === "good") good = c;
    else bad = c;
  }
  console.log(`${name}: first bad commit is ${bad}`);
  console.log(`  ${seen.join(", ")}`);
}

bisect("careless", careless);
bisect("careful", careful);
console.log(`at most ${Math.ceil(Math.log2(200))} verdicts for 200 commits, without skips`);

The careless run blames commit 95, the folder move, and never looks past it.

Here, bisection lands on a commit titled "tidy dayKey", which replaced isoDate.slice(0, 10) with the Date-based version. The earlier code never converted the string into an instant, so it could not shift the date.

Fix the cause, then the class

A root cause is a cause you can fix that explains every observation. "The report crashed" is a symptom. "The key was wrong" is closer. "Date-only strings are parsed as UTC instants and then read back with local-time getters" explains the crash on the 1st, the silent shift on other days, the timezone dependence, and why tests passed. It also tells you where else to look.

A highlighted box on the left reads root cause: UTC parse, local getters. Four arrows fan out from it to four observations stacked on the right: the crash on the 1st, the silent shift on other days, the timezone dependence, and why tests passed. A note under the root cause reads: one cause, every observation.
One sentence, date-only strings parsed as UTC and read back with local getters, accounts for all four observations. That is what makes it the root cause rather than a nearer symptom.

Any code in the repository that has to preserve a calendar date, but passes a date-only string to new Date and then calls getDate, getMonth, getDay or toLocaleDateString, has the same bug. Code that only needs the instant, or that reads it back with the UTC getters, is fine. A search for new Date( finds the candidates, and each one gets the same question. The fix for dayKey is to keep the calendar date as the string it already is. The fix for the class is that plus a CI job that runs the test suite under a timezone west of UTC, so the next person who writes the same line sees a red test.

When you review a bug fix, ask for three things: the reproduction that failed before, the root cause in one sentence that accounts for every observation, and the list of other places that sentence says the same mistake could live.

Check your understanding

8 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An invoicing job crashed with TypeError: Cannot read properties of undefined (reading 'country'). An agent changed the function to this. What happens now for an invoice whose accountId is missing from accountsById?
    TypeScript
    function taxRate(invoice: Invoice, accountsById: Map<number, Account>): number {
      const account = accountsById.get(invoice.accountId);
      if (!account) return 0;
      return ratesByCountry[account.country];
    }
    Show the answer

    Answer B. The check guards the line that failed and returns 0, so the caller gets a tax rate of zero and nothing logs that the account was missing. Whatever made the lookup miss, such as an account the loader skipped or a wrong accountId on the invoice, is still there. Returning early is right only when a missing account is a valid case, and an invoice for an account that does not exist is not one.

  2. 2An import job fails on one customer's file with Invalid date: 31/02/2026. An agent changes the job to try the parse three times in total before it gives up. What happens on the next run with the same file?
    Show the answer

    Answer A. Nothing changes between attempts: the same bytes go through the same parser and hit the same impossible date. A retry helps only when the failure depends on something that varies, like a dropped connection. Here it triples the work and delays the error, and the real question, what to do with a date that does not exist, is still open.

  3. 3You wrote a regression test, committed it on top of HEAD, and ran this. Every step was marked bad, and Git named the first commit after v2.0.0 as the culprit. Why?
    Shell
    git bisect start
    git bisect bad HEAD
    git bisect good v2.0.0
    git bisect run node --test test/totals.regress.test.ts
    Show the answer

    Answer C. Bisect checks out old commits, and a file committed only at HEAD disappears with them. Node then exits non-zero because the file is missing, which bisect reads as bad, so every step points earlier. Keep the test untracked, in a path no candidate commit touches, so it stays in place at each checkout. The first-commit answer is tempting, but no step ever tested the bug.

  4. 4There are 1,000 commits between the last good release and today, and checking one commit takes 3 minutes. Every commit builds and can be tested. About how long does bisection take?
    Show the answer

    Answer D. Each check halves the range, and 2 to the 10th is 1,024, so about 10 checks cover 1,000 commits, roughly 30 minutes. Walking the commits one by one from either end needs hundreds of checks on average. The 10 depends on every commit being testable: each commit you have to skip adds checks, and skipped commits next to the change can leave Git naming several suspects instead of one.

  5. 5You are cutting a failing 4,000-line import file down to a minimal reproduction. You delete the header row and the failure disappears. What does that tell you?
    Show the answer

    Answer A. When removing a piece makes the failure stop, that piece is needed for it, so it goes back in and you cut elsewhere. It is also a clue: something about the header shapes how later rows are read. The error naming a data row tells you where the failure showed, which can be far from the defect.

  6. 6A product page sometimes shows euro prices to US users. An agent has already made two guessed changes to the formatting code. Which next message is most likely to make progress?
    Show the answer

    Answer C. That message names a hypothesis, the prediction it makes, and a small check that could prove it wrong, so either result teaches you something. Another screenshot gets a fresh guess from the same evidence. Logging everything collects data with no question to answer, and a new library changes code before anyone knows the cause.

  7. 7A bug's root cause was: date-only strings are parsed as UTC instants and then read back in local time. On a server running in Los Angeles, which of these has the same bug?
    TypeScript
    // dueDate looks like "2026-04-01"; createdAt like "2026-04-01T09:30:00Z"
    const dueLabel = (dueDate: string) => new Date(dueDate).toLocaleDateString("en-US");
    const ageMs = (createdAt: string) => Date.now() - new Date(createdAt).getTime();
    const dueIso = (dueDate: string) => new Date(dueDate).toISOString().slice(0, 10);
    Show the answer

    Answer B. toLocaleDateString reads the instant in the machine's zone, so in Los Angeles April 1 prints as 3/31/2026. toISOString reads it back in UTC, so dueIso returns the date it was given, and ageMs only does arithmetic on instants, where the zone does not matter. The root cause has two halves, a UTC parse and a local read, and only dueLabel has both.

  8. 8A checkout test failed about once in 50 runs. An agent raised a timeout from 2 to 5 seconds, ran the suite 3 times, saw it pass every time, and called it fixed. What is missing?
    Show the answer

    Answer D. With a 1 in 50 failure, three passing runs happen about 94 percent of the time with no fix at all, so they prove almost nothing. Make the failure happen on demand, find what makes the test slow, and show the reproduction failing before the change and passing after. More clean runs shrink the doubt slowly, and a longer timeout can hide a real slowdown.

You understand it when you can

Drill

An agent 'fixed' an intermittent crash in an order-processing worker by wrapping the handler in a try/catch and adding three retries, and the crash reports stopped. Find the evidence that the cause is still present, reproduce the original failure, and name its root cause.

Topics

Reproduce First
Making a failure happen on demand, ideally as a failing test, before touching the code.
Debugging as Experiment
Forming a hypothesis, predicting what you will see, and running the smallest test that could prove the hypothesis wrong.
Bisection
Halving the search space across commits, inputs or configuration until the cause is isolated.
Reading Stack Traces and Logs
Extracting where, when and with what data a failure happened.
Debuggers and Tracing Tools
Breakpoints, watch expressions and system tracers for seeing what code actually does at run time.
Root Cause Analysis
Asking why until you reach a cause you can fix, then fixing the whole class of bug along with the instance.

Further reading

Start here

  • Debugging with the Scientific Method - Stuart Halloway. Stuart Halloway, 2015. 49-minute talk.

    Treats debugging as hypothesis, prediction and experiment, with a written record of each step, instead of repeated guessing, which is the habit the 'paste the error back to the agent' failure lacks.

Short reads

Watch

  • Lecture 4: Debugging and Profiling. MIT Missing Semester staff, 2026. 73-minute lecture.

    A hands-on 2026 MIT lecture covering debuggers, record-replay debugging, system-call tracing, bpftrace/eBPF and network debugging for seeing what code actually does at run time.

  • The Debugging Book • Andreas Zeller & Clare Sudbery • GOTO 2025. Andreas Zeller, 2025. 52-minute interview.

    The leading researcher on systematic debugging discusses automating the scientific method: delta debugging to shrink failing inputs, statistical fault localization, and tracing a failure back to its cause.

Books

  • Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems. David J. Agans, 2002.

    Nine plain rules, including 'make it fail', 'quit thinking and look', 'divide and conquer' and 'if you didn't fix it, it ain't fixed', that directly counter symptom-patching with retries and try/catch.

  • Why Programs Fail: A Guide to Systematic Debugging. Andreas Zeller, 2009, 2nd edition.

    The rigorous treatment of reproduction, simplifying failing inputs with delta debugging, scientific debugging, and tracing infection chains from symptom back to defect.

  • The Pocket Guide to Debugging. Julia Evans, 2022.

    47 concrete strategies for getting unstuck on a bug, from reading the stack trace carefully to bisecting and writing down what you know, with worked networking debugging examples.

Primary sources

  • git-bisect Documentation. Manual.

    The official manual for binary-searching commit history, including git bisect run with a test script, so the commit that introduced a regression can be found automatically.

  • Site Reliability Engineering, Chapter 12: Effective Troubleshooting. Reference.

    Google's hypothetico-deductive troubleshooting method (triage, examine, diagnose, test, treat) and its named pitfalls, such as mistaking correlation for causation and latching onto the first cause.