Building Software
Sections

Contents Section 7, Operations

Performance and Capacity

Measuring where time and money go, finding bottlenecks before users do, and planning for load.

Section 7 of 9, subsection 5 of 5. Depth: do. A 10-minute read.

Mistakes to catch in review

  1. Code optimized where it is not the bottleneck, while the real cost sits in a query or a network call.

  2. A cache with no invalidation that serves stale data, or a cache key that serves one user's data to another.

  3. A service that works for ten users and falls over at a thousand because nobody considered the connection pool or a rate limit.

  4. Autoscaling or model usage with no ceiling, turning a traffic spike into a very large bill.

Where the Time and Money Go

Your order history page takes about half a second to load. You ask an agent to make it faster. It comes back with a rewrite of formatPrice, the function that turns an integer number of cents into a string like $1,234.50.

The new version caches its Intl.NumberFormat object instead of building one per call, and a microbenchmark shows it running ten times faster. The agent reports success.

You deploy it and the page still takes half a second. The agent optimized something that was slow in isolation, without checking whether it was slow in the page. This is a common way for performance work to fail: code gets faster in a place where the request was never spending its time.

Measure the request before touching the code

When a request's stages run one after another, its total time is the sum of the time spent in each, plus any time spent waiting. (When work overlaps, the total follows the critical path, the longest chain of steps that each wait on the one before, and that chain is what you measure.) So the first job is to find out how the total splits up. You can do that with a few lines of timing code around each stage:

TypeScript
type Timings = Record<string, number>;

async function timed<T>(label: string, timings: Timings, fn: () => T | Promise<T>): Promise<T> {
  const start = performance.now();
  try {
    return await fn();
  } finally {
    timings[label] = (timings[label] ?? 0) + (performance.now() - start);
  }
}

async function orderHistory(userId: string, db: Db) {
  const timings: Timings = {};
  const orders = await timed("orders query", timings, () =>
    db.query("SELECT id, placed_at FROM orders WHERE user_id = $1", [userId]));
  for (const order of orders) {
    order.items = await timed("items queries", timings, () =>
      db.query("SELECT * FROM order_items WHERE order_id = $1", [order.id]));
  }
  const html = await timed("render", timings, () => renderOrders(orders));
  console.log(timings);
  return html;
}

For a customer with 50 orders, one run prints something like this:

Output
{ "orders query": 9.1, "items queries": 412.6, "render": 23.8 }

Total: about 445 milliseconds. formatPrice runs inside render, and all of rendering is 24 ms. Suppose price formatting was 2 ms of that and the agent made it ten times faster. The function now takes 0.2 ms, so it saved 1.8 ms out of 445, less than half a percent.

Removing the function entirely would save 2 ms, about 0.45 percent, and that is the most any change to it could save, because a speedup to one part of a program can only remove the time that part was taking. This limit is called Amdahl's law, and it is why you measure the split before you choose what to optimize.

The real cost is the loop. It sends one query per order, 50 round trips to the database, each around 8 ms including the network. This shape is called an N+1 query: one query for the list, then one more for each item in it. One query for all the items replaces all 50:

TypeScript
const ids = orders.map((o) => o.id);
const items = await db.query("SELECT * FROM order_items WHERE order_id = ANY($1)", [ids]);

The code then groups the items by order_id in memory. That query takes around 10 ms, and the page drops from about 445 ms to about 45.

You can run both shapes against a stand-in database where every query waits 8 ms. Both return the same items. The first also scans the item list once per order, but those 50 scans take well under a millisecond, so the timing shows the round trips.

TypeScriptRuns in your browser
// A stand-in database: every query is one round trip of about 8 ms.
let roundTrips = 0;
const ITEMS = Array.from({ length: 150 }, (_, i) => ({ orderId: 1 + (i % 50), sku: `sku-${i}` }));
const orderIds = Array.from({ length: 50 }, (_, i) => i + 1);

async function query<T>(rows: () => T[]): Promise<T[]> {
  roundTrips++;
  await new Promise((r) => setTimeout(r, 8));
  return rows();
}

async function itemsOneQueryPerOrder() {
  const byOrder = new Map<number, unknown[]>();
  for (const id of orderIds) {
    byOrder.set(id, await query(() => ITEMS.filter((it) => it.orderId === id)));
  }
  return byOrder;
}

async function itemsInOneQuery() {
  const wanted = new Set(orderIds);
  const rows = await query(() => ITEMS.filter((it) => wanted.has(it.orderId)));
  const byOrder = new Map<number, unknown[]>();
  for (const row of rows) byOrder.set(row.orderId, [...(byOrder.get(row.orderId) ?? []), row]);
  return byOrder;
}

for (const [label, load] of [["one query per order", itemsOneQueryPerOrder], ["one query for all", itemsInOneQuery]] as const) {
  roundTrips = 0;
  const start = performance.now();
  const byOrder = await load();
  const ms = Math.round(performance.now() - start);
  console.log(`${label}: ${byOrder.size} orders, ${roundTrips} ${roundTrips === 1 ? "query" : "queries"}, ${ms} ms`);
}

A CPU profiler would have been a poor tool for finding this. A sampling profiler records what the processor is executing, and while the code waits on the database, the process is mostly idle. The 400 ms of waiting barely appears in a CPU profile, while the 24 ms of rendering looks like the whole story.

For a request handler, start with wall-clock timing of each stage, or a tracing system that records spans for each query and outgoing call. Reach for a CPU profiler once you know the time is spent computing.

To catch a misplaced optimization, ask of any performance change what fraction of the request's time the changed code took before the change. If the agent cannot answer with a measurement, it has not found the bottleneck yet.

Averages hide the users who wait

One request tells you where the time goes. To know what users experience, you need the distribution over many requests. Latency is usually reported as percentiles. The p50 is the median; the p99 is the time that 99 percent of requests beat and 1 percent do not.

The average is a poor summary because latency distributions have long tails. A page where most requests take 40 ms and 2 percent take 3 seconds has an average near 100 ms, a number that describes no real user.

A pill at the top left, Page, calls ten in parallel. A moving teal trunk drops from it with a branch into the start of each bar in a column, so every call begins at the same moment: Service 1 and Service 2 are short, a row of dots stands for the rest, and Service 10, slow this time, is a long highlighted bar reaching far to the right. Every bar ends with an arrow into one tall box on the right, Page done, after the last call. The fast bars' arrows run a long way to reach it; the slow bar's arrow is short. A note above it reads as slow as the slowest call. Below the bars, the formula 1 − 0.99¹⁰ ≈ 9.6% reads: chance at least one of ten is slow, at 1% each.
The page waits for all ten calls, so it is as slow as the slowest one. If each service is slow on 1 percent of calls, independently, about 9.6 percent of page loads are slow.

Tails also compound. If rendering a page calls ten backend services in parallel and waits for all of them, the page is as slow as the slowest call. If each service is slow on 1 percent of calls, and the slow calls are independent, the chance that at least one of the ten is slow is 1 − 0.99¹⁰, about 9.6 percent. A 1-in-100 problem in each service becomes a nearly 1-in-10 problem for the page.

Why it works for ten users and falls over at a thousand

The second measure is throughput: how many requests per second the system completes. Latency and throughput are tied together by a relation called Little's law. The average number of requests in progress equals the arrival rate times the average time each one spends in the system.

Apply it to a resource with a fixed number of slots, like a database connection pool. Say a checkout service runs with a pool of 10 connections per instance, and each checkout request holds a connection for 50 ms. Each connection can serve 20 requests per second, so the pool can serve at most 200 requests per second.

A rounded box labeled Requests, arrive at random, flows right into a box labeled Queue, grows when arrivals pass 200 per second, which flows down into a highlighted box labeled Pool, 10 connections. An arrow labeled held 50 ms runs left from the pool to a cylinder labeled Database. Two notes under the database read: 50 ms each gives 20 per second, and 10 times 20 per second equals 200 per second.
Each of the 10 connections is held for 50 ms, so it serves 20 requests per second and the pool tops out at 200. Below that rate only bursts wait in the queue; above it the queue grows without limit.

Below that average rate, most requests find a free connection, but bursts still queue: if 15 requests arrive within the same 50 ms, 5 of them wait for a connection. Above the average rate, the queue grows without limit, and nothing about the code has changed except the load.

Queues get worse faster than intuition suggests. In the simplest queueing model, one server with random arrivals and randomly varying (exponentially distributed) service times, the average time a request spends in the system is its service time divided by one minus the utilization. A pool of ten connections is a different model, so read the numbers below as the shape of the curve, not as a prediction for that pool.

At 50 percent utilization, a 50 ms operation takes 100 ms on average. At 90 percent, 500 ms. At 95 percent, a full second. The curve is nearly flat and then nearly vertical, which is why a system can look fine at 70 percent load and collapse at 95.

The checkout pool shows the same shape. This simulation sends randomly timed requests at 10 connections, each held for 50 ms, and reports each request's wait plus its 50 ms of work. The clock is simulated, so it finishes at once.

TypeScriptRuns in your browser
const POOL = 10;
const HOLD_MS = 50;
const REQUESTS = 20_000;

let seed = 42;
function random(): number {
  // A small seeded generator, so every run prints the same numbers.
  seed = (seed * 1664525 + 1013904223) % 4294967296;
  return seed / 4294967296;
}

function simulate(perSecond: number) {
  const freeAt = new Array<number>(POOL).fill(0);
  const waits: number[] = [];
  let now = 0;
  for (let i = 0; i < REQUESTS; i++) {
    now += (-Math.log(1 - random()) / perSecond) * 1000; // random gap until the next arrival
    const slot = freeAt.indexOf(Math.min(...freeAt)); // the connection that frees up first
    const start = Math.max(now, freeAt[slot]);
    freeAt[slot] = start + HOLD_MS;
    waits.push(start - now);
  }
  const sorted = [...waits].sort((a, b) => a - b);
  const pct = (p: number) => Math.round(sorted[Math.floor(p * (sorted.length - 1))] + HOLD_MS);
  return { p50: pct(0.5), p99: pct(0.99), lastWait: Math.round(waits[waits.length - 1]) };
}

for (const rate of [50, 100, 150, 180, 190, 195, 210]) {
  const r = simulate(rate);
  console.log(`${rate}/s  p50 ${r.p50} ms  p99 ${r.p99} ms  last request waited ${r.lastWait} ms`);
}

Up to 150 per second the median request does not wait at all. At 195, which is 97.5 percent of capacity, the p99 is more than ten times the work itself. At 210 the arrivals outpace the pool, and the wait keeps growing for as long as the traffic lasts.

With ten users, none of this is visible. The pool is idle most of the time. That is why these limits are found in production unless you look for them on purpose.

The same arithmetic applies to every fixed limit on the path. Postgres ships with max_connections set to 100. If autoscaling runs 15 instances, each with a pool of 10, they will try to open 150 connections and some will be refused.

A third-party API with a limit of 600 requests per minute supports 10 per second across your whole fleet, however many instances you run. When you review an agent's design, list every pool, limit and quota on the request path, and divide.

Load testing: finding the knee

A load test sends traffic at a controlled rate and records latency and errors. Raise the rate in steps and hold each step long enough for the numbers to settle. Here is what that looks like for one instance of the checkout service:

Requests/sec p50 p99 Errors
50 58 ms 110 ms 0%
100 60 ms 130 ms 0%
150 68 ms 240 ms 0%
190 140 ms 900 ms 0%
220 2.1 s timeout 12%

The degradation point, often called the knee, is between 150 and 190. Past it, latency climbs quickly while throughput stops growing.

The table shows where the service degrades. To find the cause, check each resource the request uses for three things: how busy it is (utilization), how much work is waiting for it (saturation), and whether it is returning errors. Brendan Gregg calls this the USE method.

Here the database CPU sits at 35 percent, which rules out CPU saturation and nothing else: lock waits, slow storage or a single saturated core could still limit it. So check those as well. In this test, lock waits stay near zero and each query still takes about 12 ms at the knee, so the database is not what slows down. The application's CPU is at 40 percent. The pool's wait time, the time requests spend waiting for a free connection, rises from zero to hundreds of milliseconds right at the knee. The bottleneck is the connection pool, and its ceiling matches the 200 per second from the arithmetic.

Two panels. The top panel, labeled Before, connection held 50 ms, contains three boxes in a row: Open transaction, then a highlighted Payment call, over HTTP, connection idle, then Write result. The bottom panel, labeled After, shorter hold, contains Open transaction then Write result; outside it on the left, a highlighted box labeled Payment call, outside the transaction, points into the panel with a moving teal arrow. A note reads: higher ceiling, no new connections.
Before, the connection sits idle inside the transaction while the payment call waits on HTTP, so each request holds it for 50 ms. With the call moved outside the transaction, the hold is shorter and the pool's ceiling rises without adding a connection.

Reading the handler shows why each request holds a connection for 50 ms when its queries take 12: it opens a transaction, calls a payment provider over HTTP, then writes the result. The connection sits idle during the HTTP call. Moving the call outside the transaction shortens the hold time and raises the ceiling without adding a single connection.

Many load generators send a request, wait for the response, then send the next. When the server slows down, such a generator slows its own sending rate, so it never produces the queue a real crowd would, and it under-reports latency at the tail.

Gil Tene named this effect coordinated omission. Configure the tool to send at a fixed arrival rate whatever the server does. In k6 that is the constant-arrival-rate executor.

Caches: every input belongs in the key

Once you know where the time goes, a cache is often the fix. A cache stores the result of a computation under a key, so the next request with the same key skips the work. It is correct only when two requests with the same key would have produced the same result. So the key must include every input the result depends on.

Keys go wrong because many inputs are implicit. Take an endpoint that quotes shipping for a product:

TypeScript
async function shippingQuote(productId: string, viewer: { country: string; tier: "member" | "free" }) {
  const key = `shipping:${productId}`;
  const hit = await cache.get(key);
  if (hit) return hit;
  const quote = await computeShipping(productId, viewer.country, viewer.tier);
  await cache.set(key, quote, { ttlSeconds: 300 });
  return quote;
}

The key has one input and the computation has three. The first visitor after the cache expires decides the quote everyone sees for the next five minutes. If that visitor was a paid member in Germany, customers in the United States see free shipping priced in euros.

Here is the same mistake with a plain map as the cache, next to a key that holds every input.

TypeScriptRuns in your browser
type User = { country: "DE" | "US"; tier: "member" | "free" };
type Quote = { cents: number; currency: "EUR" | "USD" };

const cache = new Map<string, Quote>();

function computeShipping(productId: string, country: User["country"], tier: User["tier"]): Quote {
  const cents = tier === "member" ? 0 : country === "DE" ? 690 : 850;
  return { cents, currency: country === "DE" ? "EUR" : "USD" };
}

function cached(key: string, compute: () => Quote): Quote {
  const hit = cache.get(key);
  if (hit) return hit;
  const quote = compute();
  cache.set(key, quote);
  return quote;
}

const keyOnProduct = (productId: string, user: User) =>
  cached(`shipping:${productId}`, () => computeShipping(productId, user.country, user.tier));

const keyOnEveryInput = (productId: string, user: User) =>
  cached(`shipping:${productId}:${user.country}:${user.tier}`, () => computeShipping(productId, user.country, user.tier));

const memberInGermany: User = { country: "DE", tier: "member" };
const visitorInUS: User = { country: "US", tier: "free" };

for (const [label, quote] of [["key on product only", keyOnProduct], ["key on every input", keyOnEveryInput]] as const) {
  cache.clear();
  const first = quote("lamp-42", memberInGermany);
  const second = quote("lamp-42", visitorInUS);
  console.log(`${label}: member in Germany pays ${(first.cents / 100).toFixed(2)} ${first.currency}, visitor in the US pays ${(second.cents / 100).toFixed(2)} ${second.currency}`);
}

To audit a cache, read the function it wraps and list everything it reads: its arguments, plus the current user, their session, locale, country, permissions, feature flags and the clock. Every item on that list must be in the key, or must provably not change the output. When the output belongs to one user, either the user's ID goes in the key or the response stays out of any shared cache.

HTTP caches follow the same rule. A response marked Cache-Control: private may be stored by the user's browser but not by a CDN or proxy. Vary tells a cache which request headers to add to its key. A page carrying a user's account details that is served without either can be stored once and handed to the next visitor.

The other half of correctness is staleness. A cached value is a copy, and the original can change. Deleting the cache entry when the data is written helps, but it has a race.

Two horizontal lanes, Reader on top and Writer below, with time running left to right and the steps numbered. 1, in the Reader lane: Cache miss, reads the old value. A long arrow labeled old value in hand runs along the Reader lane past the next two steps. Under it, a bracket marked meanwhile drops dashed legs down to 2 and 3 in the Writer lane: Update row, new value in database, then Delete key, cache entry removed. 4, back in the Reader lane: highlighted Store value, old value cached. A dashed arrow drops from it to a pill labeled Expiry, the known bound, and a note beside the pill reads only the expiry bounds staleness.
The reader fetches the old value before the writer updates the row, then stores it after the writer has deleted the key. Nothing removes that stale entry, so only an expiry puts a bound on how long it stays.

A reader misses the cache and reads the old value from the database. A writer updates the row and deletes the key. The reader then stores its old value in the cache, where it stays. For that reason, every cache entry should carry an expiry even when you also invalidate on write, so that the worst-case staleness has a known bound.

Put a ceiling on cost

Capacity costs money, and some capacity scales itself. An autoscaler with no maximum answers a traffic spike, or a bot, by adding instances until the bill stops it. A feature that calls a language model does the same thing per request.

Suppose each call costs one cent. A scraper hitting that endpoint 50 times a second makes 4.32 million calls a day, which is $43,200 a day.

Treat spend like latency: measure it per feature, set a budget, and alert when it moves. Then put a hard ceiling in front of every resource that scales with traffic, such as a maximum instance count, a per-user rate limit on model calls, and a spending cap with the provider where one exists.

For any design an agent hands you, compute two numbers before it ships: the request rate at which the first fixed limit on its path is exhausted, and the most it can cost in one hour if traffic goes to a hundred times normal.

Check your understanding

8 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An agent timed each stage of a slow product page and got this. It then made the template engine four times faster. About how much faster is the page?
    Output
    { "auth": 3.2, "load cart": 6.1, "recommendations": 388.4, "render": 41.0 }
    Show the answer

    Answer C. Render drops from 41 ms to about 10, which saves about 31 ms of a 439 ms request. The 75 percent figure is the speedup of render alone. Almost all of the request is the 388 ms recommendations stage, so that is the stage where a change could pay off.

  2. 2This handler takes about a second. Each database round trip costs about 5 ms. A CPU profile of the process shows JSON serialization at the top. What is the best next step?
    TypeScript
    async function feed(userId: string) {
      const notes = await db.query(
        "SELECT id, actor_id, kind FROM notifications WHERE user_id = $1 LIMIT 200", [userId]);
      for (const n of notes) {
        const [actor] = await db.query("SELECT name, avatar FROM users WHERE id = $1", [n.actor_id]);
        n.actor = actor;
      }
      return notes;
    }
    Show the answer

    Answer B. Up to 200 lookups run one after another at about 5 ms each, which is the second. While the process waits on them it is idle, so a CPU profile barely shows them, and a faster serializer only speeds up the small slice the profile can see. users.id is almost certainly the primary key already, and an index does not reduce the number of round trips.

  3. 3An agent added this cache to the shared catalog page. What has to change for the cache to be correct?
    TypeScript
    // Set by the app's routing and auth middleware before the handler runs.
    type CatalogRequest = {
      query: { category: string; page: number };
      user: { id: string; locale: string };
    };
    
    async function catalogPage(req: CatalogRequest) {
      const key = `catalog:${req.query.category}:${req.query.page}`;
      const hit = await cache.get(key);
      if (hit) return hit;
      const showBeta = flags.isOn("beta-catalog", req.user);
      const rows = await listProducts(req.query.category, req.query.page, { includeBeta: showBeta });
      const html = renderCatalog(rows, req.user.locale);
      await cache.set(key, html, { ttlSeconds: 600 });
      return html;
    }
    Show the answer

    Answer D. The page depends on four inputs: category, page, whether this user sees beta products, and the user's locale. Two are missing from the key, so the first visitor's flag and language go to everyone for ten minutes. A flag is evaluated per user, so it is not the same for everyone, and a shorter TTL only shortens how long each wrong page is served.

  4. 4Each checkout request holds one of the instance's 20 database connections for 80 ms. What is the most checkouts per second one instance can complete?
    Show the answer

    Answer A. One connection held for 80 ms serves 1 / 0.08 = 12.5 requests per second, and 20 connections serve 250. The 1,600 comes from multiplying 20 by 80, which treats a duration as if it were a rate. 12.5 is the ceiling of a single connection, not of the pool.

  5. 5A partner API allows 1,200 requests per minute for your account. Each of your 6 instances limits itself to 5 calls per second. What happens at peak traffic?
    Show the answer

    Answer B. The limit belongs to the account, so it covers the whole fleet: 1,200 per minute is 20 per second in total, and six instances at 5 per second send 30. Each instance looks fine on its own, which is why you list every limit on the path and divide it across the instances that share it.

  6. 6An agent ran this load test from a machine in the same data center as the service and reports a p99 of 180 ms. Production times the same thing at the load balancer, from sending the request to the service until the last byte of the response comes back, and at a similar average rate its p99 is 2 seconds. What most likely explains the gap?
    TypeScript
    // 50 virtual users, each sending its next request as soon as the last one returns
    async function virtualUser(results: number[]) {
      while (Date.now() < endAt) {
        const start = Date.now();
        const res = await fetch(TARGET);
        await res.arrayBuffer(); // wait for the whole body, as production's timer does
        results.push(Date.now() - start);
      }
    }
    await Promise.all(Array.from({ length: 50 }, () => virtualUser(results)));
    Show the answer

    Answer A. Each virtual user waits for its response before sending again, so a slow response delays the next request. Real users keep arriving whatever the server does, which builds the queue this script never creates, and the slow periods are under-sampled. This is coordinated omission; a fixed arrival rate fixes it. Distance to real users is not in either number, because both are timed inside the same data center.

  7. 7A service deletes a cache key whenever its row changes and sets no expiry, since every write invalidates. A reader misses the cache and reads the row. A writer updates the row and deletes the key. The reader then stores what it read. How long can that stale value be served?
    Show the answer

    Answer C. The reader's write to the cache lands after the writer's delete, so the old value stays with nothing to remove it until another write. The delete did run after the update, but the stale value was stored after both. With no expiry set, there is no default to fall back on; an expiry on every entry is what bounds this.

  8. 8An agent shipped a summarize endpoint that calls a model at $0.002 per call, behind an autoscaler with no maximum. A bot calls it 30 times a second around the clock. Which change puts a ceiling on the bill?
    Show the answer

    Answer D. At 30 calls a second the bot costs about $5,184 a day. An alert tells you about it and stops nothing, and a cache does nothing when the bot varies its input. A ceiling has to refuse work: a rate limit per user, a maximum instance count, and a cap the provider enforces.

You understand it when you can

Drill

An agent sped up a slow product page by memoizing a pricing function, and added a cache keyed only on the product ID for a response that includes the viewer's saved cart. Profile to find where the page actually spends its time, and find the cache bug that shows one user another user's cart.

Topics

Latency, Throughput and Tail Latency
The two basic performance measures, and why the slowest percentiles decide user experience.
Profiling
Measuring CPU, memory and I/O to find where time actually goes before optimizing anything.
Load Testing and Capacity Planning
Simulating realistic traffic to find limits, and planning resources ahead of growth.
Caching and Invalidation
Where caches help, how they go stale, and designing keys and expiry so they stay correct.
Cost as a Metric
Tracking compute, storage and model spend per feature, with budgets and alerts like any other signal.

Further reading

Start here

  • "Performance Matters" by Emery Berger. Emery Berger, 2019. 42-minute talk.

    Shows how layout effects and noise mislead naive benchmarks, and introduces causal profiling (Coz) to find the code whose speedup would actually make the program faster.

Short reads

Watch

  • "How NOT to Measure Latency" by Gil Tene. Gil Tene, 2015. 43-minute talk.

    Explains why averages and even p99 hide what users experience, and how coordinated omission makes most load-testing tools under-report tail latency.

  • Web Cache Deception Attack. Omer Gil, 2017. 23-minute talk.

    Shows real sites whose caches stored one user's authenticated page and served it to others because the cache key and rules ignored who was asking, the per-user caching bug in this subsection's drill.

Books

  • Systems Performance: Enterprise and the Cloud. Brendan Gregg, 2020, 2nd edition.

    The standard reference for performance methodology (USE method, workload characterization) and for profiling CPU, memory, disk and network before changing any code.

  • Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. Martin Kleppmann and Chris Riccomini, 2026, 2nd edition.

    Its opening chapters define load, throughput, latency percentiles and tail-latency amplification, and later chapters explain how caches, replicas and derived data go stale.

  • The Art of Capacity Planning: Scaling Web Resources in the Cloud. Arun Kejariwal and John Allspaw, 2017, 2nd edition.

    Shows how to measure real capacity ceilings from production data, forecast growth, and plan resources and autoscaling limits before traffic arrives.

Primary sources

  • The Tail at Scale (Dean and Barroso, Communications of the ACM, 2013). Paper.

    Shows mathematically why rare slow responses dominate user-facing latency once a request fans out to many servers, and describes hedged requests and other ways to reduce the tail.

  • RFC 9111: HTTP Caching. RFC.

    The HTTP caching standard, defining cache keys, Vary, Cache-Control private and no-store, and freshness, the rules that decide whether a shared cache may store and reuse a per-user response.