Building Software

Engineering Fundamentals for the Agent Era

Contents Section 7, Operations

Debugging

Mistakes to catch in review

  1. A symptom patched with a null check, a retry or a broad try/catch while the actual cause stays in place.

  2. A fix declared successful without the original failure ever being reproduced.

  3. The same error pasted back to an agent again and again while nobody forms a hypothesis about the cause.

Finding the cause of a failure by reproducing it and running experiments, instead of guessing at fixes.

Topics

Reproduce First
Making a failure happen on demand, ideally as a failing test, before touching the code.
Debugging as Experiment
Forming a hypothesis, predicting what you will see, and running the smallest test that could prove the hypothesis wrong.
Bisection
Halving the search space across commits, inputs or configuration until the cause is isolated.
Reading Stack Traces and Logs
Extracting where, when and with what data a failure happened.
Debuggers and Tracing Tools
Breakpoints, watch expressions and system tracers for seeing what code actually does at run time.
Root Cause Analysis
Asking why until you reach a cause you can fix, then fixing the whole class of bug along with the instance.

You understand it when you can

  • Reduce a failing case to a minimal reproduction.
  • Use bisection to find the commit that introduced a regression.
  • Explain a bug's root cause well enough to predict where else the same mistake appears.

Drill

An agent 'fixed' an intermittent crash in an order-processing worker by wrapping the handler in a try/catch and adding three retries, and the crash reports stopped. Find the evidence that the cause is still present, reproduce the original failure, and name its root cause.

Start here

Watch

Debugging with the Scientific Method - Stuart Halloway

Stuart Halloway, 2015. 49-minute talk.

Treats debugging as hypothesis, prediction and experiment, with a written record of each step, instead of repeated guessing, which is the habit the 'paste the error back to the agent' failure lacks.

Watch

Lecture 4: Debugging and Profiling

MIT Missing Semester staff, 2026. 73-minute lecture.

A hands-on 2026 MIT lecture covering debuggers, record-replay debugging, system-call tracing, bpftrace/eBPF and network debugging for seeing what code actually does at run time.

Read

Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems

David J. Agans, 2002.

Nine plain rules, including 'make it fail', 'quit thinking and look', 'divide and conquer' and 'if you didn't fix it, it ain't fixed', that directly counter symptom-patching with retries and try/catch.

Why Programs Fail: A Guide to Systematic Debugging

Andreas Zeller, 2009, 2nd edition.

The rigorous treatment of reproduction, simplifying failing inputs with delta debugging, scientific debugging, and tracing infection chains from symptom back to defect.

The Pocket Guide to Debugging

Julia Evans, 2022.

47 concrete strategies for getting unstuck on a bug, from reading the stack trace carefully to bisecting and writing down what you know, with worked networking debugging examples.

Primary sources

  • Manual

    git-bisect Documentation

    The official manual for binary-searching commit history, including git bisect run with a test script, so the commit that introduced a regression can be found automatically.

  • Reference

    Site Reliability Engineering, Chapter 12: Effective Troubleshooting

    Google's hypothetico-deductive troubleshooting method (triage, examine, diagnose, test, treat) and its named pitfalls, such as mistaking correlation for causation and latching onto the first cause.