Building Software
Sections

Contents Section 7, Operations

Observability

Logs, metrics and traces that let you answer new questions about a running system and learn it is broken before users tell you.

Section 7 of 9, subsection 2 of 5. Depth: do. Core 17 of 25. A 9-minute read.

Mistakes to catch in review

  1. Whole request bodies logged, including passwords, tokens and personal data.

  2. Errors logged without a request ID, inputs or cause, so they can't be traced or reproduced.

  3. Alerts on CPU and memory while user-facing errors climb unnoticed.

  4. A user ID used as a metric label, creating millions of time series and a large monitoring bill.

Questions You Can Answer From Outside

A customer writes in: "Checkout failed at about two o'clock, and then I saw the charge on my card." You open the checkout service's logs and find this, written by code an agent added last month:

Output
2026-09-30T14:02:08Z INFO checkout {"email":"dana@example.com","cardNumber":"4111111111111111","cvc":"123","items":[...]}
2026-09-30T14:02:11Z ERROR payment failed

The first line contains a full card number, which now sits in every system that stores or forwards these logs. The second line says something failed and nothing else: which order, which request, how long it waited, what the payment service said. You cannot tell whether the error line belongs to this customer's request, and you cannot tell whether they were really charged.

A running service is a closed box. Once it is deployed, everything you will ever know about what it did comes from what it wrote down while doing it. Observability is the practice of making it write down the right things, so that when a question comes up that you did not plan for, the answer is already in the data.

Three kinds of record

There are three shapes of data a service can emit, and each answers a different kind of question.

An event is a record of one thing that happened: one request, one job run, one failed login. It can carry as many details as you like, because you write it once and read it rarely. Logs are events.

A metric is a number summarised over time: requests per second, errors per minute, the 99th percentile of latency. A metric throws away the details of individual events and keeps a running count, which makes it cheap to store for months and fast to graph and alert on.

A trace is a set of linked events describing one request as it passes through several services. Each step is a span with a start time, a duration and a pointer to the span that called it. Put together, the spans show where the time went and which hop failed.

The mistakes in this subsection come from putting the wrong data in the wrong shape: details that belong in events stuffed into metrics, or events written with too little detail to be useful.

Logs as records

A log line written as a sentence, like payment failed for order 99107 after 3004ms, can be read by a person and searched only with fragile text matching. A log line written as a record with named fields can be filtered, grouped and counted: all errors for order_id=ord_99107, all requests where duration_ms > 2000, the error rate per route. That is what structured logging means. The format is usually JSON, one object per line.

The most useful pattern is one wide event per unit of work. Instead of scattering ten small lines through a request handler, collect fields as the request proceeds and write one line at the end with everything: who, what, how long, what happened, and the error if there was one.

TypeScript
import { randomUUID } from "node:crypto";

type Fields = Record<string, string | number | boolean | null>;

function log(level: "info" | "error", event: string, fields: Fields): void {
  console.log(JSON.stringify({ ts: new Date().toISOString(), level, event, ...fields }));
}

export async function handleCheckout(req: Request): Promise<Response> {
  const requestId = req.headers.get("x-request-id") ?? randomUUID();
  const started = performance.now();
  const fields: Fields = { request_id: requestId, route: "POST /checkout" };

  try {
    let body: any;
    try {
      body = await req.json();
    } catch {
      fields.status = 400;
      fields.error = "malformed JSON";
      return new Response("bad request", { status: 400 });
    }
    if (
      typeof body?.orderId !== "string" ||
      typeof body.userId !== "string" ||
      !Number.isInteger(body.amountCents) ||
      !Array.isArray(body.items)
    ) {
      fields.status = 400;
      fields.error = "invalid request body";
      return new Response("bad request", { status: 400 });
    }

    // Allowlist: name each field that goes in the log. The card token and email never do.
    Object.assign(fields, {
      user_id: body.userId,
      order_id: body.orderId,
      amount_cents: body.amountCents,
      item_count: body.items.length,
    });

    const pay = await fetch(process.env.PAYMENTS_URL + "/charge", {
      method: "POST",
      headers: { "content-type": "application/json", "x-request-id": requestId },
      body: JSON.stringify({ orderId: body.orderId, amountCents: body.amountCents, cardToken: body.cardToken }),
      signal: AbortSignal.timeout(3000),
    });
    fields.payments_status = pay.status;
    await pay.body?.cancel(); // only the status is needed; release the connection
    if (!pay.ok) throw new Error(`payments returned ${pay.status}`);
    fields.status = 200;
    return new Response("ok");
  } catch (err) {
    fields.status = 502;
    fields.error = err instanceof Error ? `${err.name}: ${err.message}` : String(err);
    return new Response("payment failed", { status: 502 });
  } finally {
    fields.duration_ms = Math.round(performance.now() - started);
    log(Number(fields.status) >= 500 ? "error" : "info", "checkout", fields);
  }
}

The request ID and the base fields are set before anything is parsed, the parsing happens inside the try, and the log call sits in finally. Every way out of the function, including a malformed body that is rejected with a 400, writes exactly one event. A handler that parsed the body first would log nothing for a request that failed to parse, and those are the requests you most need to see.

The allowlist comes next. The code names each field that goes into the log, so a new field added to the request body next month stays out until someone decides it belongs. The opposite approach, logging the body and redacting known secret fields, fails the first time someone adds a field the redaction list does not know about. Run both approaches on today's request body and on next month's, after a saved-card feature adds a token to it:

TypeScriptRuns in your browser
type Body = Record<string, unknown>;

// Redaction: log everything except the fields someone remembered to list.
const SECRET_FIELDS = ["password", "cardNumber", "cvc"];
function redactingLogLine(body: Body): string {
  const copy: Body = { ...body };
  for (const key of SECRET_FIELDS) if (key in copy) copy[key] = "[redacted]";
  return JSON.stringify(copy);
}

// Allowlist: log only the fields someone decided belong in the log.
function allowlistLogLine(body: Body): string {
  return JSON.stringify({ order_id: body.orderId, amount_cents: body.amountCents });
}

const today: Body = { orderId: "ord_99107", amountCents: 4599, cardNumber: "4111111111111111", cvc: "123" };
const nextMonth: Body = { ...today, savedCardToken: "tok_8f2a91c4" };

console.log("today, redaction:     ", redactingLogLine(today));
console.log("today, allowlist:     ", allowlistLogLine(today));
console.log("next month, redaction:", redactingLogLine(nextMonth));
console.log("next month, allowlist:", allowlistLogLine(nextMonth));

Both lines are safe today. Next month the redacting logger writes the saved-card token in full, although nobody touched the logging code. The allowlist line has not changed.

A highlighted box on the left, one log line with a session token, sends four moving teal arrows into a frame labeled kept for weeks. Inside the frame are four boxes: log shippers, search clusters, vendor dashboards, and incident laptops. A note under the log line reads readable by more people.
One log line fans out to every place logs are copied, and each copy is kept for weeks by more people. A token written once is a credential in all of them.

Logs are copied to more places than your database: log shippers, search clusters, vendor dashboards, laptops during an incident. They are kept for weeks and readable by more people. A password or session token in a log is a credential anyone with log access can use. When you review logging code, look for any line that passes a whole request, response, headers object or user record to the logger.

The error line needs context as well. The failure line carries the request ID, the order, the amount, the payment service's status if it answered, the error's type and message, and how long the request took. The test for an error log is whether the line identifies the request and the failing operation, and points to the data needed to reproduce the failure safely: the request ID finds the request in other systems, and the order ID finds the stored order you can replay against a test environment. The line should not be a recipe for re-sending the request, because that could charge the card a second time. ERROR payment failed fails that test. So does a line that includes the message but not the inputs, because "timeout" alone does not tell you which order or which dependency.

Following one request across two services

Look at the x-request-id header in the code above. Checkout takes the ID from the incoming request if a caller already set one, makes one otherwise, logs it, and sends it on to the payment service, which logs it too. That shared value is a correlation ID, and it is what turns two separate piles of logs into one story.

Back to the customer. You search checkout's logs for their user_id around 14:02 and find one line:

JSON
{"ts":"2026-09-30T14:02:11.482Z","level":"error","event":"checkout","request_id":"7f3c9a1e-5d2b-4c1a-9e7f-2b8d6a4c0f13","route":"POST /checkout","user_id":"u_18422","order_id":"ord_99107","amount_cents":4599,"item_count":2,"status":502,"error":"TimeoutError: The operation was aborted due to timeout","duration_ms":3004}

Then you search the payment service's logs for that request_id:

JSON
{"ts":"2026-09-30T14:02:11.611Z","level":"info","event":"charge","request_id":"7f3c9a1e-5d2b-4c1a-9e7f-2b8d6a4c0f13","order_id":"ord_99107","status":200,"processor_ms":2987,"duration_ms":3120}

The two lines together explain the complaint. The payment service charged the card successfully, but it took 3.12 seconds, mostly waiting on the card processor. Checkout gave up at 3 seconds and told the customer it failed. Neither log alone shows that. The fix is a design question (a longer timeout, or checking the charge status before reporting failure), and you know which question to ask because the IDs lined up.

Three bars on one time axis marked 0 ms, 1,000 ms and 2,000 ms. The top bar, checkout span, 502 after 3,004 ms, ends at 3 seconds. Below it a highlighted bar, payment span, 200, charged, 3,120 ms, starts just inside checkout and runs past checkout's end. Below that, processor call, 2,987 ms, fills most of the payment bar. A dashed line drops from a note reading checkout gives up, at the end of the checkout bar, to 3,004 ms on the axis; the payment bar crosses it. A note beside the payment bar's end reads about 130 ms later, payment carried on, and child outlasts its parent.
Drawn as spans on one time axis, the payment call starts inside checkout and ends about 130 ms after checkout gave up at 3 seconds. The child outlasting its parent is the evidence that the card was charged after the customer saw a failure.

Distributed tracing does the same job with more structure. Each request gets a trace ID, each unit of work inside it gets a span ID, and each span records its parent. The IDs travel between services in a header; the W3C Trace Context standard names it traceparent and packs a version, the 32-hex-digit trace ID, the 16-hex-digit parent span ID and some flags into one value. A tracing backend then draws the request as a timeline, and the payment service's span would show as a bar about 3.1 seconds long, most of it the 2,987 ms processor call. That bar starts inside the checkout span but ends about 130 ms after the checkout span does, since checkout stopped waiting at 3 seconds and payment carried on. A child span can outlast its parent, and here that is the evidence that checkout timed out while the charge went through. OpenTelemetry is the common open-source way to produce these spans and to put the trace ID into your log lines, so logs and traces can be joined.

Metrics and the cost of a label

A metric is stored as a set of time series. A series is one metric name plus one exact combination of label values, like http_requests_total{route="POST /checkout", status="502"}. Every distinct combination is its own series, stored and indexed separately, and hosted metrics services commonly price by how many series you have.

So the number of series is the product of the number of values each label can take. With 20 routes, 8 status codes and 3 regions, a request counter has at most 480 series. Add a user_id label and the ceiling becomes 480 times the number of users. With 2 million active users that is up to 960 million series for one counter, and every new user adds more. Monitoring systems slow down or run out of memory long before that, and the bill grows with every sign-up.

This simulation sends 200,000 requests from up to 50,000 users and counts the series a counter creates with and without a user_id label:

TypeScriptRuns in your browser
const routes = ["GET /cart", "POST /checkout", "GET /orders/:id"];
const statuses = ["200", "400", "502"];

// A small seeded random generator, so every run prints the same numbers.
let seed = 42;
function random(n: number): number {
  seed ^= seed << 13; seed ^= seed >>> 17; seed ^= seed << 5;
  return (seed >>> 0) % n;
}

const withoutUser = new Set<string>();
const withUser = new Set<string>();

for (let i = 1; i <= 200_000; i++) {
  const route = routes[random(routes.length)];
  const status = statuses[random(statuses.length)];
  const user = "u_" + random(50_000);
  withoutUser.add(`route=${route},status=${status}`);
  withUser.add(`route=${route},status=${status},user_id=${user}`);
  if (i === 1_000 || i === 20_000 || i === 200_000) {
    console.log(`after ${i} requests: ${withoutUser.size} series without user_id, ${withUser.size} with it`);
  }
}

Without the label the count stops at 9, one per route and status pair. With it, most requests create a new series, and the count keeps climbing toward 9 × 50,000 = 450,000. In production the user count itself keeps growing, so there is no ceiling at all.

Two rows. Top: fixed values, routes × statuses × regions, flow into metric labels, cost per series. Bottom: highlighted unbounded values, `user_id` and order IDs, flow into event fields, cost per volume. A dashed arrow rises from unbounded values and ends at a large cross marked no ceiling, just under the metric labels box. A note reads per-user questions go to logs.
Values from a small fixed set make good labels, because series count is the product of their values. Unbounded IDs belong in events, where cost follows traffic volume rather than the number of distinct users.

Labels must have a small, fixed set of possible values. User IDs, email addresses, order IDs, request IDs and raw URLs with IDs in the path all have unbounded values, and they belong in events, where each record carries its own details without creating a new series. Storing and indexing events still costs money, so what you pay for is volume, which grows with traffic and not with the number of distinct users. If you need per-user questions answered, query the logs.

For choosing which metrics a service should have, two short lists cover most cases. For anything that handles requests, measure rate, errors and duration (the RED method). For resources like CPUs, disks and connection pools, measure utilization, saturation and errors (the USE method). Rate comes from a counter, a number that only goes up, read as its increase per second. The number of open connections in a pool is a gauge, a value that goes up and down. Duration is a histogram, which counts requests into latency buckets so you can compute percentiles later.

Defining "working" from the user's side

A service level indicator (SLI) is a measurement of what users experience, written as a ratio: good events divided by valid events. For checkout, a reasonable SLI is the fraction of checkout requests that did not return a 5xx status. A second one could be the fraction that completed within one second.

A service level objective (SLO) is the target for that ratio over a window, for example 99.9% of checkout requests succeed over 30 days. The remaining 0.1% is the error budget. At 10 checkouts per second, 30 days is 25,920,000 requests, so the budget is 25,920 failed checkouts. In time, 0.1% of 30 days is 43.2 minutes of total outage.

Alerting then becomes a question about the budget. The burn rate is how fast you are spending it relative to the pace that would use it up exactly at the end of the window. A burn rate of 1 spends the budget in 30 days. A burn rate of 14.4 sustained for one hour spends 14.4 / 720 = 2% of the whole month's budget, since 30 days is 720 hours and, at a steady 10 checkouts per second, each hour holds 1/720 of the month's requests. That is a reasonable threshold for waking someone: an hour-long error rate above 14.4 × 0.1% = 1.44%.

PromQL
sum(rate(http_requests_total{route="POST /checkout", status=~"5.."}[1h]))
  /
sum(rate(http_requests_total{route="POST /checkout"}[1h]))
  > 0.0144

Practical setups pair this with a shorter window, such as five minutes, that must also be over the threshold, so the page stops soon after the problem does. The SRE Workbook chapter on alerting on SLOs works through that design.

Compare this with an alert on CPU above 80%. Suppose the payment service's credentials expire and every charge is rejected immediately. Checkout now returns 502 in a few milliseconds instead of doing real work, so its CPU goes down. The CPU alert stays quiet while every customer fails. The burn-rate alert fires within minutes, because it measures the thing users feel. High CPU with happy users is something to look at during working hours, and a failing checkout at low CPU is an outage. Page on symptoms users experience, and keep resource metrics on dashboards and in tickets.

Review observability code against a customer complaint you invent. The logs should take you from their words to the one failing request without guessing, and an alert on a user-facing SLI should have fired before they wrote in.

Check your understanding

7 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An agent added this log line to a profile-update endpoint. What should the reviewer flag first?
    TypeScript
    app.post("/api/profile", async (req, res) => {
      log.info("profile update", { userId: req.user.id, headers: req.headers, body: req.body });
      await profiles.update(req.user.id, req.body);
      res.sendStatus(204);
    });
    Show the answer

    Answer A. Request headers hold the session cookie or bearer token, and the body may hold anything the client sends next month. Encryption at rest protects the disk, while every person and tool that reads the logs still sees the token in plain text. Dropping to debug level only hides the problem until someone turns debug on during an incident. A user ID in an event is useful and expected; the credentials are the leak.

  2. 2This service handles 200 orders a second. On-call sees 40 of these error lines in one minute. Which change lets them diagnose a failure from the log alone?
    TypeScript
    try {
      await inventory.reserve(order.id, order.items);
    } catch (err) {
      log.error("reserve failed");
      throw err;
    }
    Show the answer

    Answer C. The line has to say which request failed, on what input and why. The request ID finds it in other services' logs, the order ID finds the stored data, and the error and duration say whether it was a timeout or a rejection. A stack trace alone says where the code was and nothing about which of the 40 orders it was. Logging the whole order copies addresses and payment details into the logs, and an order ID label makes a new metric series per order.

  3. 3After this change ships, the number of time series for http_requests_total grows every day. Why?
    TypeScript
    // Express middleware. req.path is the URL path as received, such as /orders/ord_99107.
    app.use((req, res, next) => {
      res.on("finish", () => {
        httpRequests.inc({ route: req.path, status: String(res.statusCode) });
      });
      next();
    });
    Show the answer

    Answer D. A series is one exact combination of label values. The raw path puts an unbounded value, the order ID, into the route label, so each new order creates another series. The fix is to label with the route pattern, like /orders/:id. Calling inc more often only makes an existing series count higher; it does not add series, which is why per-request cost is not the problem here.

  4. 4An API's SLO is 99.5% of requests succeed over 30 days. Traffic is steady, the same number of requests every hour of the month. For one hour, 7% of requests fail. What is the burn rate, and how much of the month's error budget did that hour use?
    Show the answer

    Answer B. The budget is 0.5% of requests, so failing 7% spends it 0.07 / 0.005 = 14 times faster than the pace that would last the month. Thirty days is 720 hours, and with steady traffic each hour holds 1/720 of the month's requests, so one hour at that rate spends 14 / 720, about 1.9% of the budget. A busier hour would spend more. The tempting 14% reads the burn rate as a share of the budget and forgets that the hour is only one of 720.

  5. 5Your checkout SLO is 99.9% of requests succeed over 30 days. Which of these should page someone at 3am?
    Show the answer

    Answer C. A 3% failure rate against a 0.1% budget is a burn rate of 30, more than twice the 14.4 that spends about 2% of the month's budget in an hour when traffic is steady, and customers are failing right now. High CPU with healthy errors and latency hurts no one yet, so it belongs on a dashboard or in a ticket. The disk trend can wait for a ticket in working hours.

  6. 6The orders service calls inventory with a 2 second timeout. These are the two log lines for one request. Where did the time go?
    JSON
    {"service":"orders","request_id":"r_51c0","status":504,"error":"TimeoutError: inventory call timed out","duration_ms":2004}
    {"service":"inventory","request_id":"r_51c0","status":200,"duration_ms":48}
    Show the answer

    Answer B. Inventory measured 48 ms from the moment its handler started to the moment it finished, so its database was not the slow part. Orders waited 2 seconds and gave up, so nearly all of that time passed before inventory's handler started or while the response travelled back. The durations differ because each service measures its own part. Orders timed out before the response arrived, so its own code after the response never ran.

  7. 7Which requests to this handler leave no log line at all?
    TypeScript
    export async function handleRefund(req: Request): Promise<Response> {
      const started = performance.now();
      const body = await req.json();
      const fields: Record<string, unknown> = { order_id: body.orderId, route: "POST /refund" };
      try {
        await payments.refund(body.orderId, body.amountCents);
        fields.status = 200;
        return new Response("ok");
      } catch (err) {
        fields.status = 502;
        fields.error = String(err);
        return new Response("refund failed", { status: 502 });
      } finally {
        fields.duration_ms = Math.round(performance.now() - started);
        log("refund", fields);
      }
    }
    Show the answer

    Answer A. req.json() runs before the try, so a malformed body throws before the try starts, and its finally never runs. A failing refund is caught inside the try, and finally logs it with the error. Moving the parse inside the try, after the base fields are set, gives every way out of the function exactly one event.

You understand it when you can

Drill

An agent added logging to a login endpoint that writes the full request body at info level, emits a login metric labeled with each user's ID, and alerts when CPU exceeds 80 percent. Find the data leak, the metrics-cost problem and the outage this alerting would miss.

Topics

Structured Logging
Machine-readable log events with consistent fields, useful context and no secrets.
Metrics
Counters, gauges and histograms, and the RED and USE methods for choosing what to measure.
Distributed Tracing
Following one request across services with trace and span IDs to see where time and errors go.
SLIs, SLOs and Error Budgets
Defining reliability from the user's side and deciding how much failure is acceptable.
Symptom-Based Alerting
Paging people when users are hurting, and keeping every alert actionable.

Further reading

Start here

Short reads

  • My Philosophy on Alerting. Rob Ewaschuk.

    Page on symptoms users feel, not on causes, and every page should need a human now.

  • Canonical Log Lines. Brandur Leach.

    One wide log line per request with everything you would want to query later.

  • Logs vs Structured Events. Charity Majors.

    Why arbitrary-width events answer questions you did not know to ask when you wrote the code.

Watch

Books

  • Observability Engineering: Achieving Production Excellence. Charity Majors, Liz Fong-Jones and 2 others, 2026, 2nd edition.

    A near-complete rewrite that covers structured events, tracing, SLO-based alerting and telemetry cost, and adds material on validating AI-generated code in production.

  • Implementing Service Level Objectives: A Practical Guide to SLIs, SLOs, and Error Budgets. Alex Hidalgo, 2020.

    Walks through choosing SLIs from the user's point of view, setting targets, and using error budgets and burn rates to decide when to page and when to slow down.

  • Learning OpenTelemetry: Setting Up and Operating a Modern Observability System. Ted Young and Austin Parker, 2024.

    Explains how OpenTelemetry ties traces, metrics and logs together through shared context propagation, and how to instrument services and run the Collector in practice.

Primary sources

  • Alerting on SLOs (The Site Reliability Workbook, Chapter 5). Reference.

    Works through six alerting strategies step by step, ending at multiwindow, multi-burn-rate alerts that page when the error budget is burning, where CPU thresholds only fire on symptoms users may never feel.

  • Prometheus: Metric and label naming. Manual.

    The official warning that every unique label combination is a new time series and that labels must never hold unbounded values such as user IDs, which is the metrics-cost trap in this subsection's drill.