Building Software
Sections

Contents Section 4, Math and Algorithms

Probability and Statistics

Distributions, sampling, variance and significance: the tools for judging whether a measurement, an experiment or an eval result means anything.

Section 4 of 9, subsection 3 of 4. Depth: explain. A 9-minute read.

Mistakes to catch in review

  1. Average latency reported while a slow tail, which most users hit at least once per session, stays hidden.

  2. A prompt or model change declared an improvement from a handful of runs, with no estimate of run-to-run variance.

  3. Failures treated as independent when they share a cause, such as two replicas in the same data center.

  4. A flaky test called fixed after a single green run.

Measuring Things That Vary

An agent rewrites the system prompt for your support bot, runs the eval suite, and reports back: pass rate up from 80% to 84%, average response latency down from 72 ms to 66 ms, and the flaky integration test that failed now and then is fixed because it just passed. Each of those sentences is a measurement of something that varies, and each one can be true while meaning nothing.

The suite has 50 cases, so 80% versus 84% is 40 passes versus 42. The average hides whatever the slowest requests are doing. One green run of a flaky test is what a flaky test produces most of the time. Probability and statistics are the tools for asking how much a number would move by chance, and whether the difference in front of you is bigger than that.

The average hides the requests people remember

Take 1,000 requests to a service. Suppose 950 of them take 40 ms, 45 take 300 ms because they missed a cache, and 5 take 3,000 ms because they hit a garbage-collection pause or a retry.

The mean is the total divided by the count: (950 × 40 + 45 × 300 + 5 × 3,000) / 1,000 = 66.5 ms. No request took 66.5 ms. The number mixes a big fast group with a small slow group and describes neither.

The median is the middle value when the requests are sorted, here 40 ms. It tells you what a typical request feels like. A percentile generalizes that: the 99th percentile, written p99, is the value that 99% of requests are at or below. Sort the 1,000 times, and the 990th is 300 ms. The p99.9 is the 999th, which is 3,000 ms.

TypeScriptRuns in your browser
function percentile(sorted: number[], p: number): number {
  const rank = Math.ceil((p / 100) * sorted.length); // nearest-rank method
  return sorted[Math.max(rank, 1) - 1];
}

const times = [
  ...Array(950).fill(40),
  ...Array(45).fill(300),
  ...Array(5).fill(3000),
].sort((a, b) => a - b);

const mean = times.reduce((sum, t) => sum + t, 0) / times.length;
console.log(mean);                    // 66.5
console.log(percentile(times, 50));   // 40
console.log(percentile(times, 99));   // 300
console.log(percentile(times, 99.9)); // 3000
Three groups of requests sized by count: 950 requests at 40 ms each, 45 at 300 ms from cache misses, and 5 at 3,000 ms from a pause or retry. Below them, the median at 40 ms points to the 950, p99 at 300 ms to the 45, and p99.9 at 3,000 ms to the 5. The mean, 66.5 ms, sits between the groups because no request took it.
The median and the high percentiles each land on a real group of requests. The mean, 66.5 ms, falls between groups where no request took that long.

Latency distributions usually look like this: a large cluster of fast responses and a long tail of slow ones caused by cache misses, queueing, pauses and retries. The mean does react to the tail, but it blends the tail into one number with everything else, so you cannot tell from it whether five requests took three seconds or fifty took a little longer. The median does not see the tail at all. The high percentiles show the tail that the mean and median hide. So when an agent reports an improvement in average latency, ask for p50, p99 and p99.9 before and after. A change that makes the fast requests faster and the slow ones slower can lower the average and make the product worse.

Why the tail is the common case for a user

A user does not make one request. Loading a page, scrolling a feed or finishing a checkout can take dozens. If each request independently has a 1% chance of landing above the p99, the chance that a session of k requests hits at least one of them is 1 − 0.99^k.

For 10 requests that is about 10%. For 50 it is about 40%. For 100 it is about 63%. Past roughly 70 requests, more than half of sessions include at least one request slower than the p99. That is why the p99 describes something most users actually feel, even though its name suggests it only affects 1 in 100. The same arithmetic applies inside a system: a request that fans out to 100 servers and waits for all of them is slower than the p99 of a single server most of the time. Dean and Barroso's paper "The Tail at Scale" works through this in detail.

Independence is an assumption about causes

That calculation multiplied probabilities, and multiplying is only correct when the events are independent: one happening tells you nothing about whether the other happens. The same assumption sits under most reliability estimates, and it is where designs go wrong.

One power feed, which fails 0.05% of the time, sits above Replica A and Replica B, each down 0.1%. Both replicas hang from the feed, and both lead to Both down: at least 0.05%, not the 0.0001% an independence assumption gives.
One power feed under both replicas takes both down at once, so the pair is down at least 0.05% of the time, not 0.0001%.

Suppose an agent proposes two database replicas, each unavailable 0.1% of the time, and reports that both will be down at once only 0.1% × 0.1% = 0.0001% of the time, about 30 seconds a year. That number assumes the replicas fail for unrelated reasons. If both sit in the same data center on the same power feed, a single power failure takes out both. If that feed fails 0.05% of the time, the pair is down at least 0.05% of the time, about 4.4 hours a year, at least 500 times the independent estimate. Shared causes include a data center, a power feed, a network switch, a bad deploy pushed to both, and a certificate that expires on both at the same minute.

The question that exposes the assumption: what single event would make both of these fail? If you can name one, the probabilities do not multiply, and the joint failure rate is at least the rate of that event.

How much a number moves by itself

Go back to the eval. Treat each of the 50 cases as a pass-or-fail draw with an 80% chance of passing. The standard error of a pass rate, which measures how far it typically lands from the true rate, is

√(p × (1 − p) / n) = √(0.8 × 0.2 / 50) ≈ 0.057,

or about 5.7 percentage points. The difference between two independent runs has a standard error √2 times larger, about 8 points. A rough rule is that an observed difference needs to be around two standard errors of the difference before it stands out from noise. On 50 cases that is about 16 points: a measured gap smaller than that is a result chance produces often. The rule sets what a suite of this size can detect, and the true change can be smaller than that and still real. A 4-point gain is two cases flipping, well inside what chance produces.

Top row: 50 cases, each passing 80%, give one run a standard error of about 5.7 points; times root 2, the difference between two runs is about 8 points; times 2, a difference stands out at about 16 points. Bottom row: the observed gain, 80% to 84%, two cases, is 4 points. The 16-point bar and the 4-point gain meet at one comparison, 4 < 16, which leads to Inside the noise.
On 50 cases a difference needs about 16 points to stand out from chance. The agent's 4-point gain is two cases flipping, inside the noise.

The formula is a first estimate. Language models are also nondeterministic, so the same prompt on the same case can pass on one run and fail on the next. The direct way to measure that is to run the old version against itself. Run version A on the suite five times and look at the spread of the five pass rates. If A varies between 76% and 86% with nothing changed, a result of 84% for version B tells you nothing about B.

The playground simulates that with a 50-case suite in which every case passes 80% of the time and nothing ever changes. Its random numbers come from a fixed seed, so it prints the same result on every run.

TypeScriptRuns in your browser
// A small seeded random number generator (mulberry32).
function seeded(seed: number) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const random = seeded(7);

function passRate(cases: number, p: number): number {
  let passed = 0;
  for (let i = 0; i < cases; i++) if (random() < p) passed++;
  return Math.round((100 * passed) / cases);
}

const fiveRuns = Array.from({ length: 5 }, () => passRate(50, 0.8));
console.log("version A, five runs:", fiveRuns.map((r) => r + "%").join(", "));

const pairs = 20_000;
let gains = 0;
for (let i = 0; i < pairs; i++) if (passRate(50, 0.8) - passRate(50, 0.8) >= 4) gains++;
console.log(`one run of A beat another run of A by 4+ points in ${Math.round((100 * gains) / pairs)}% of pairs`);

The same version scores anywhere from 72% to 84% across five runs. One run of it beats another by at least the agent's 4 points about a third of the time, with nothing changed.

To make a small difference detectable, you need more cases. For a 4-point gap near 80% to reach two standard errors, √(2 × 0.16 / n) has to be at most 0.02, which gives n of at least 800 cases per version. Even then, if the true improvement is exactly 4 points, the measured gap lands above or below 4 about equally often, so a real gain clears the bar only about half the time. Small improvements need large suites, repeated runs, or both. Comparing the two versions case by case on the same inputs helps, because the difficulty of each case then cancels out of the comparison.

A streak of greens is weak evidence

A flaky test is one that fails some fraction of runs for the same code. Suppose a test failed about one run in five before an agent's fix. After the fix it passes once, and the agent calls it fixed.

Ask how likely that green run would be if the fix did nothing. If the failure rate is still one in five, each run passes with probability 0.8. One green run happens 80% of the time. A streak of n green runs, assuming runs are independent, happens 0.8^n of the time:

green runs in a row chance of that streak if the failure rate is still 1 in 5
1 80%
3 51%
5 33%
10 11%
14 4.4%

To get that chance below 5%, solve 0.8^n < 0.05. Take logarithms: n > ln(0.05) / ln(0.8) ≈ 13.4, so 14 consecutive green runs. If the failure rate had not changed, a streak of 14 would come up 4.4% of the time, so it counts as evidence against the old rate. The probability that the bug remains is a different number, and it also depends on how likely the fix was to work before you ran anything.

The number of runs has to be fixed in advance, for example the next 14, with every result recorded. If you keep rerunning until a streak of 14 shows up, the 4.4% no longer applies, because you gave chance many tries to produce it. The rarer the original failure, the longer the streak you need, because a test that failed 1 run in 50 produces long green streaks all by itself.

The playground tries both habits on 10,000 tests whose fix did nothing. One plan decides on 14 runs before it starts. The other reruns, up to 100 times, and stops as soon as it sees 14 greens in a row.

TypeScriptRuns in your browser
// A small seeded random number generator (mulberry32).
function seeded(seed: number) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const random = seeded(1);

// The fix did nothing: every run still fails one time in five.
const runIsGreen = () => random() >= 0.2;

function decideRunsFirst(): boolean {
  for (let i = 0; i < 14; i++) if (!runIsGreen()) return false;
  return true;
}

function rerunUntilStreak(): boolean {
  let streak = 0;
  for (let i = 0; i < 100; i++) {
    streak = runIsGreen() ? streak + 1 : 0;
    if (streak === 14) return true;
  }
  return false;
}

const tests = 10_000;
for (const plan of [decideRunsFirst, rerunUntilStreak]) {
  let calledFixed = 0;
  for (let i = 0; i < tests; i++) if (plan()) calledFixed++;
  console.log(`${plan.name}: ${((100 * calledFixed) / tests).toFixed(1)}% of still-broken tests called fixed`);
}

Deciding first calls about 4% of the broken tests fixed, close to the 4.4% the arithmetic predicts. Rerunning until the streak appears calls about 60% of them fixed.

The independence assumption matters here too. If the test fails only when the machine is under load or when the clock is near midnight, fifty green runs in a row on a quiet laptop prove little. Rerun it under the conditions where it used to fail.

Rare events make accurate detectors noisy

Say a fraud detector catches 99% of fraudulent transactions and wrongly flags 1% of legitimate ones. It sounds 99% accurate. Now suppose 1 transaction in 1,000 is fraud, and run 100,000 transactions through it.

There are 100 fraudulent transactions, and the detector flags 99 of them. There are 99,900 legitimate ones, and it flags 1% of those, which is 999. So it raises 1,098 flags, and only 99 of them are fraud. When the detector fires, it is right about 9% of the time.

100,000 transactions split into fraud, 100 of them (1 in 1,000), and legitimate, 99,900. The detector catches 99 of the frauds, 99%, and wrongly flags 999 legitimate ones, 1%. Both streams join in 1,098 flags, of which only 99 are fraud, about 9%.
The detector catches 99 of 100 frauds but also flags 999 legitimate transactions. Of 1,098 flags only 99 are fraud, about 9%.

This is Bayes' rule at work. The chance that a flag is real depends on how accurate the detector is and on how common the thing it detects is, called the base rate. When the base rate is small, the false positives from the huge pool of normal cases outnumber the true positives from the tiny pool of real ones, even with a good detector. The same arithmetic applies to security alerts, anomaly detection, classifiers that screen content, and a monitor that pages you on a rare failure. When an agent quotes a detector's accuracy, ask for three numbers: the base rate b, the fraction s of real cases the detector catches, and the fraction f of legitimate cases it flags anyway. For N events, true alerts are N × b × s and false alerts are N × (1 − b) × f. Here that is 100,000 × 0.001 × 0.99 = 99 and 100,000 × 0.999 × 0.01 = 999, and 99 of 1,098 flags is the 9% above.

A measurement means something only when you know how much it would vary with nothing changed. Before accepting a difference, estimate or measure that variation and check that the difference is at least twice its own standard error.

Check your understanding

6 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An agent measured ten requests and reports: average latency 251.8 ms, so the service is slow for everyone. What is wrong with that reading?
    TypeScript
    const ms = [12, 15, 11, 14, 13, 12, 16, 11, 14, 2400];
    const mean = ms.reduce((a, b) => a + b, 0) / ms.length;
    console.log(mean); // 251.8
    Show the answer

    Answer A. No request took anywhere near 251.8 ms. The median is about 13 ms, which is what a typical request feels like, and the one 2,400 ms request is the tail a user meets now and then. Deleting it hides exactly the slow requests that p99 and p99.9 exist to show. The arithmetic itself is correct.

  2. 2An agent estimates how often both regions of a service are down at once. Both regions resolve through the same DNS provider, which is down 0.05% of the time. What is the least the pair is down?
    TypeScript
    // Each region is unavailable 0.2% of the time.
    const bothDown = 0.002 * 0.002; // 0.0004% of the time
    Show the answer

    Answer B. Multiplying assumes the regions fail for unrelated reasons. A DNS outage takes out both at once, so the pair is down at least as often as the shared provider is, 0.05% of the time. Adding the two region rates gives a bound on how often at least one region is down, which is a different question.

  3. 3Prompt v1 scored 88% on a 25-case eval suite. The agent's rewrite, v2, scored 92% on one run, and the agent wants to ship it as the better prompt. What should happen first?
    Show the answer

    Answer C. On 25 cases, 88% versus 92% is 22 passes versus 23, a single case. Running v1 against itself shows how far the score moves with nothing changed, and a gap inside that spread says nothing about v2. A second run of v2 alone still has no baseline for the noise, and more cases shrink the noise for both.

  4. 4A login detector catches 95% of account takeovers and flags 2% of normal logins. Takeovers are 1 login in 10,000. Of the logins it flags, roughly what share are takeovers?
    Show the answer

    Answer C. In 1,000,000 logins there are 100 takeovers, and it flags 95 of them. It also flags 2% of the 999,900 normal logins, about 20,000. So 95 of roughly 20,100 flags are real, under half a percent. The 95% says how many takeovers get flagged, and the question asks how many flags are takeovers. With a rare base rate, false alarms from the huge pool of normal logins swamp the true ones.

  5. 5A test failed only when the CI machine was under heavy load. After a fix, an agent ran it 50 times in a row on an idle laptop, all green, and calls it fixed. What is the problem?
    Show the answer

    Answer D. Streak arithmetic treats each run as an independent draw with the old failure rate. Here the failure depended on load, and on an idle laptop the chance of failing may be near zero whether or not the bug is fixed. Rerun the test under load, on the machines where it used to fail, before trusting the streak.

  6. 6An agent checks a flaky-test fix with the loop below and reports that it passed. Suppose the fix did nothing and the test still fails one run in five. What is wrong with the check?
    TypeScript
    let greens = 0;
    while (greens < 5) {
      greens = (await runTest()) ? greens + 1 : 0;
    }
    console.log("fixed: 5 green runs in a row");
    Show the answer

    Answer D. The loop has no limit on attempts, so a test that passes 80% of the time eventually produces five greens in a row and the loop always prints fixed. Raising the target to 14 only makes it take longer. The number of runs has to be fixed before the first one, with every result counted.

You understand it when you can

Drill

An agent reports a flaky test as fixed after three consecutive green runs; before the fix it failed about one run in ten. Compute the chance of three greens with the bug still present, and say how many consecutive green runs it takes before a bug that is still there would produce that streak less than 1 percent of the time.

Topics

Probability Basics
Events, independence, conditional probability and expected value.
Distributions and Percentiles
Normal and long-tailed distributions, and why percentiles describe user experience better than averages.
Sampling and Variance
How much a measurement moves between runs, what a confidence interval says, and how many samples are enough.
Experiments and Significance
A/B tests, hypothesis testing and the common ways experiments fool the people who run them.
Base Rates and False Positives
Why rare events make detectors, alerts and classifiers noisy, and how Bayes' rule corrects intuition.

Further reading

Start here

  • The medical test paradox, and redesigning Bayes' rule. Grant Sanderson, 2020. 21-minute explainer.

    Works through why a highly accurate test for a rare condition produces mostly false positives, which carries over directly to flaky-test reruns, alerts and classifiers.

Short reads

Watch

  • "How NOT to Measure Latency" by Gil Tene. Gil Tene, 2015. 43-minute talk.

    Shows why averages and naive percentiles hide the slow tail that most users hit, and how coordinated omission corrupts latency measurements.

  • But what is the Central Limit Theorem? Grant Sanderson, 2023. 31-minute explainer.

    Builds intuition for how sample means spread and shrink with sample size, the basis for confidence intervals and for deciding how many runs are enough.

Books

  • Think Stats: Exploratory Data Analysis. Allen B. Downey, 2025, 3rd edition. Free to read online.

    Teaches distributions, percentiles, estimation and hypothesis testing through runnable notebooks, so the ideas turn into code a developer can apply to their own measurements.

  • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Ron Kohavi, Diane Tang and Ya Xu, 2020.

    Covers the ways experiments mislead people in practice, including underpowered tests, peeking and invalid randomization, drawn from running A/B tests at Microsoft, Google and LinkedIn.

  • Introduction to Probability. Joseph K. Blitzstein and Jessica Hwang, 2019, 2nd edition.

    The Harvard Stat 110 text on conditional probability, independence, expectation and distributions, for readers who want the foundations behind the rules of thumb.

Primary sources