Building Software
Sections

Contents

Section 7 of 9

Operations

Running Software in Reality

5 subsections, the core path through it Core 16–18.

Why it matters when agents do the typing

Production is where software meets real users, real data and real failures, and someone has to answer when it breaks. Debugging and observability tell you what is happening and why, redundancy lets the system survive what nobody predicted, and safe change lets a team ship many times a day.

The course: 5 lessons, about 47 minutes of reading, in the order below.

  1. 7.1

    Debugging

    Core 16 · Depth: do

    Finding the cause of a failure by reproducing it and running experiments, instead of guessing at fixes.

    A mistake it teaches you to catch: A symptom patched with a null check, a retry or a broad try/catch while the actual cause stays in place.

    Lesson: Make It Fail, Then Ask Why 9 min

  2. 7.2

    Observability

    Core 17 · Depth: do

    Logs, metrics and traces that let you answer new questions about a running system and learn it is broken before users tell you.

    A mistake it teaches you to catch: Whole request bodies logged, including passwords, tokens and personal data.

    Lesson: Questions You Can Answer From Outside 9 min

  3. 7.3

    Reliability, Redundancy and Recovery

    Depth: do

    Designing systems that survive component failures, recovering data and service when they don't, and learning from every incident.

    A mistake it teaches you to catch: Backups that were never restored and turn out to be empty, partial or corrupt when they are needed.

    Lesson: Green Checks and Real Failures 9 min

  4. 7.4

    Shipping Change Safely

    Core 18 · Depth: do

    Version control, deployment pipelines, migrations, feature flags and rollback: the machinery for changing a live system many times a day.

    A mistake it teaches you to catch: A migration that locks a large table, or drops a column the running version still reads.

    Lesson: Two Versions at Once 10 min

  5. 7.5

    Performance and Capacity

    Depth: do

    Measuring where time and money go, finding bottlenecks before users do, and planning for load.

    A mistake it teaches you to catch: Code optimized where it is not the bottleneck, while the real cost sits in a query or a network call.

    Lesson: Where the Time and Money Go 10 min

Section test: Operations

10 questions across the 5 lessons above. Take it after the course; each answer explains itself.

  1. 1Checkout returns 502 on about 1 percent of requests, and an agent has already tried two different fixes. You group the last hour of checkout's wide events by currency and get this. What is the next step?
    Output
    currency  requests  status 502
    USD         41,200           0
    EUR         12,950           0
    GBP          3,310           0
    JPY            598         598
    Show the answer

    Answer C. Every JPY checkout fails and no other currency does, so the failure is deterministic and tied to one input. A retry would run the same defect again, and a longer timeout guesses at a cause the data does not point to. The wide events already narrowed the search to one field, so the next move is a reproduction that fails on demand, which later proves the fix. A user_id label would add a series per customer and still not say why JPY fails.

  2. 2Each instance has a pool of 10 database connections, and each request holds one for 50 ms, so one instance can complete about 200 requests per second. Traffic reaches 230 per second per instance. The readiness check below takes its connection from the same pool. What happens?
    TypeScript
    export async function readyz(pool: Pool): Promise<Response> {
      try {
        await withTimeout(pool.query("SELECT 1"), 500);
        return new Response("ok");
      } catch {
        return new Response("not ready", { status: 503 });
      }
    }
    Show the answer

    Answer B. Above 200 per second the arrivals outpace the pool, so the wait for a free connection keeps growing for as long as the traffic lasts; a simulation of this pool passes 500 ms within a few seconds. The check waits in that same queue, and every instance has the same pool and the same load, so they all fail together. Moving traffic off one instance only adds it to the others. What happens next depends on the load balancer's behavior when every target is unhealthy. The fix is capacity: shorter connection holds, more connections within the database limit, or shedding load.

  3. 3Midway through an expand-and-contract change to price_cents, the column exists and every writer fills both columns. While the backfill is still running, the change below goes to a 5 percent canary. The canary's 5xx rate and p99 match the old version. What is the canary doing?
    TypeScript
    // rows: SELECT p.price, p.price_cents, li.quantity FROM line_items li
    //   JOIN products p ON p.id = li.product_id WHERE li.order_id = $1
    const totalCents = rows.reduce((sum, r) => sum + r.price_cents * r.quantity, 0);
    Show the answer

    Answer A. Rows the backfill has not reached still hold null in price_cents, and JavaScript turns null into 0 in arithmetic, so those lines add nothing to the total. Nothing throws, so a comparison of errors and latency sees a healthy canary while it undercharges. Two lessons meet here: switching reads belongs after the backfill finishes, and a canary needs a check on what users get, such as order totals per version, because an error rate only counts crashes.

  4. 4A release will change the cached product entry from { price } in dollars to { priceCents }, under the same key. The cache answers 95 percent of product reads, entries live 24 hours, and the old version reads entries as shown. Which plan lets you roll back at any point without errors and without sending those reads to the database?
    TypeScript
    // old version, the one a rollback brings back
    const cached = await cache.get(`product:${id}`);
    if (cached) return renderPrice(cached.price); // renderPrice calls price.toFixed(2)
    Show the answer

    Answer D. An entry written as { priceCents } has no price, so the restored code calls toFixed on undefined and throws until that entry expires. Flushing avoids the error by sending every product read to the database at once, 20 times its usual read load at a 95 percent hit rate. A one-minute TTL lowers the hit rate for the whole rollout and still fails for that minute. Writing both fields is expand and contract applied to cached data: each version finds the field it reads, and price goes only when rollback is off the table. A versioned key such as product:v2:${id} also avoids the errors, but the new version starts with a cold cache.

  5. 5During a traffic spike, Postgres refuses checkout's new connections, and the errors stop when traffic falls. The setup is below. Which change fixes the cause?
    YAML
    autoscaling:
      min_instances: 4
      max_instances: null   # no ceiling
    app:
      db_pool_size: 10      # connections per instance
    postgres:
      max_connections: 100
    Show the answer

    Answer C. Each instance may open 10 connections, and Postgres keeps 3 of its 100 slots for superusers by default, so from 10 instances the fleet can fill every slot it may use, and at 15 it wants 150. A bigger pool reaches the limit sooner. A retry hits the same full server and only delays the error, the symptom patch the debugging lesson warns about. A readiness check would pull new instances out without adding capacity. Dividing the fixed limit by the per-instance pool gives the ceiling; 9 instances leaves room for other clients, and an alert at 80 percent of connections warns before the next spike.

  6. 6The nightly backup script now uses set -euo pipefail and exits 1 when the dump is too small. It runs from cron, which mails failures to a local mailbox nobody reads. Which check tells someone that backups have stopped, before they need one?
    Show the answer

    Answer B. A job that fails loudly into a mailbox nobody reads is as silent as one that reports success. A log line that stops appearing pages no one, because alerts fire on what is present, and a dashboard helps only when someone looks. Alerting on the age of the newest good backup checks the outcome you need whatever broke: the credential, cron, the disk or the script. Tying it to the restore check, and not to a file existing, catches a dump that is empty or will not load; to catch one whose contents are old, the restore check should also confirm that the newest row in the restored data is recent.

  7. 7A new read path ships behind a flag, turned on for 5 percent of checkout traffic. Every request on that path fails, and nobody notices for an hour. Traffic is steady through the month, and the checkout SLO is 99.9 percent success over 30 days. How much of the month's error budget did that hour spend, and what is the fastest way to stop it?
    Show the answer

    Answer D. A 5 percent error rate against a 0.1 percent budget is a burn rate of 50. One hour at that rate spends 50 / 720 of the month's budget, since 30 days is 720 hours, which is about 6.9 percent. The flag separates releasing the behavior from deploying the code, so turning it off stops the failures in seconds, with no deploy and no question of what a rollback leaves behind. An hour-window burn-rate alert at 14.4 would have fired after about 17 minutes.

  8. 8One cache sits in front of two kinds of lookup. A product miss costs the database about 2 ms, and a search miss about 400 ms. This week the cache's single hit-rate number fell from 98 to 95 percent, and database load rose about fourfold. Split by lookup kind, the numbers are below. What happened?
    Output
    lookup     requests/s   hits/s, last week   hits/s, this week
    product         9,500                9,400               9,400
    search            500                  400                 100
    all            10,000                9,800               9,500
    Show the answer

    Answer A. A hit rate counts lookups, and these lookups are not equal. Product misses stayed at 100 a second, about 0.2 seconds of database work each second. Search misses rose from 100 to 400, from about 40 to 160 seconds of work each second, which is the fourfold rise. The overall rate moved only 3 points because search is 5 percent of lookups. Labeling the counter by lookup kind, a small fixed set of values, and measuring database time per query would have shown the expensive slice on its own.

  9. 9At 15:00 a release's migration dropped orders.discount_code, which the old code reads. The only remaining copy is last night's 02:00 dump, and orders kept arriving all day. Which recovery plan loses the least?
    Show the answer

    Answer C. A rollback redeploys code and leaves the schema as the release left it, so the old code still finds no column, and a down migration brings the column back empty. Restoring production from the dump returns the column but throws away 13 hours of every other write, which is what a nightly dump's recovery point means. Restoring to a scratch database and copying only the lost column back keeps today's orders. The codes written between 02:00 and 15:00 existed nowhere else, which is why a drop belongs at the end of expand and contract, after nothing reads the column.

  10. 10Between v3.1 and v3.2, a search endpoint's p50 went from 90 ms to 95 ms and its p99 from 120 ms to 300 ms. There are 180 commits in between. An agent starts bisection with this check. What goes wrong?
    Shell
    # check.sh, run as: git bisect run ./check.sh
    ms=$(./time-request.sh "/search?q=lamp")   # one request, in milliseconds
    [ "$ms" -lt 200 ]
    Show the answer

    Answer B. The regression lives in the tail. On a bad commit the median request takes about 95 ms, so a single request usually passes the 200 ms check, and bisection trusts each verdict without revisiting it. A few wrong good marks send it into the wrong half. The command must fail only for the bug you are chasing, which for a latency regression means measuring the p99 over many requests per commit and exiting 125 when the result is too noisy to call. About 8 verdicts cover 180 commits, so a slower, honest check is affordable.