Building Software
Sections

Contents Section 8, Security

Sensitive Data and Secrets

Knowing which data is sensitive and which is not, where it travels, and how to keep the sensitive set small.

Section 8 of 9, subsection 3 of 6. Depth: do. Core 20 of 25. An 8-minute read.

Mistakes to catch in review

  1. API keys or passwords hardcoded in source code or committed to the repository.

  2. Personal data sent to a third-party analytics service or model provider that has no need for it.

  3. Tokens or personal data placed in URLs, where they end up in server logs, browser history and referrer headers.

  4. Test fixtures and screenshots built from real customer data.

Where One Phone Number Ends Up

You asked an agent to add text-message delivery updates to your shop. It added a phone column to the users table, a field on the account page and a call to an SMS provider. Customers get a text when their parcel ships, and the feature works.

Look at the rest of the diff. The request logger now records every form body, so each phone number lands in your logs. The analytics call sends the whole user object as "traits," which includes email, phone and date of birth. The support chat, which asks a language model to summarize each conversation, now pastes the full user record into the prompt. The phone-verification link is https://shop.example/verify?phone=15551234567&code=829301. There is a new fixtures/users.json with twenty rows the agent exported from production "for realistic test data." And the SMS provider's API key sits in src/sms.ts.

The phone field, typed on the account page, sits in the middle. Arrows to the left store it in the users table and send it through the SMS provider, marked as needed for the feature. Arrows to the right reach a group of five copies nobody needed: the request log, analytics with the whole user as traits, the support-chat model prompt, the verification URL with the number in the query, and the fixture file fixtures/users.json. A delete request removes the row from the users table, and a note says the other copies stay.
The feature needs the users table and the SMS provider; the other five copies came along with the change, and deleting the customer's row leaves them where they are.

None of these copies were needed for the feature. Access and retention rules differ between these copies. When a customer asks you to delete their data, you delete one row, and the other copies stay where they are.

Sorting data by what a leak would cost

The first step is to decide which data is sensitive. A classification scheme sorts every field into a small number of classes by one question: who is harmed if this leaks, and how badly? A common four-class scheme has these classes:

Class Meaning Examples
Public Already published; a leak costs nothing Product names, prices, public display names
Internal For staff only; a leak is embarrassing but harms no individual Internal record IDs, feature flags, order counts
Confidential Identifies or describes a person; a leak harms them Email, phone, address, order history
Restricted A leak enables direct harm such as account takeover or fraud, or carries specific legal duties Password hashes, API keys, card numbers, government IDs, health records

Handling rules attach to the class, so a field starts with default rules from its label instead of a fresh judgment call. Restricted fields are never logged or copied into test data, and they go only to the one service whose job requires them, such as a payment processor for card numbers. Confidential fields go only to services that need them for a stated purpose. The label supplies the defaults. Reassess it when the field's use or the data around it changes, and check each destination's need on its own.

The shop's users table, classified field by field:

Field Class Reason
id Internal Meaningless alone, though it links to everything else about the user
display_name Public Shown next to the user's product reviews
email Confidential Identifies the person and is their login name
phone Confidential Identifies the person; if it is used for account recovery, treat it as restricted
date_of_birth Confidential Combined with name and address, it helps someone impersonate the user
password_hash Restricted A copy lets an attacker guess passwords offline
last_login_ip Confidential Reveals roughly where the person is and can count as personal data under laws such as the EU's GDPR
created_at Internal Harmless alone

Some data carries legal duties on top of this. Depending on who you are and where your users live, health information can fall under HIPAA in the United States, card numbers under the PCI DSS standard, data about people in the EU under GDPR, and data collected from children under 13 in the US under COPPA. Record the legal and contractual obligations that apply to a field next to its sensitivity label, and use the stricter handling while someone who knows the law assesses them.

Tracing one field

Classification tells you which fields to worry about. Tracing tells you where they are. Pick one field, phone, and follow it from the moment the customer types it.

  1. The users table in the primary database.
  2. The database's read replica and its nightly backups, which keep deleted rows until each backup expires.
  3. A cache that stores the user profile for faster page loads.
  4. The request log, shipped to a log service where every engineer can search it for 90 days.
  5. The analytics provider, through the identify call.
  6. The SMS provider, which needs it to send the text.
  7. The model provider, through the support-chat prompt.
  8. The verification URL, which can be recorded in web-server, CDN or reverse-proxy access logs and in the customer's browser history.
  9. The fixture file, which is in every clone of the repository and stays in git history after the file is deleted.

The database stores the number and the SMS provider needs it. The replica and backups hold it because they copy the database. The profile cache may not need it at all, so check, and omit it if it does not. The other five copies, in the logs, analytics, the model prompt, the URL and the fixture file, are ones the agent created while making something else work. Error reports and crash dumps belong on this list too whenever they capture request data, so a trace has to include every service the application talks to and every place it writes.

Closing the leaks

Start with the logger. The agent's version records everything.

TypeScript
app.use((req, _res, next) => {
  log.info("request", { method: req.method, url: req.originalUrl, body: req.body });
  next();
});

req.originalUrl includes the query string, so the verification code and phone number from the URL are logged too. The fix logs the path without the query string, and it builds the logged body from values the application has checked.

TypeScript
app.use((req, _res, next) => {
  log.info("request", { method: req.method, path: req.path, body: loggableBody(req.body) });
  next();
});

There are three common ways to write loggableBody. A denylist removes the names it knows are sensitive. An allowlist of names keeps only approved fields. The third keeps approved fields and logs each value only after checking it against what the server knows. Run all three on two request bodies that hide a phone number and on one ordinary order:

TypeScriptRuns in your browser
const url = "/verify?phone=15551234567&code=829301";
const bodies: Record<string, unknown> = {
  "nested and renamed": { productId: { phone: "15551234567" }, quantity: 2, mobile: "15551234567" },
  "phone as productId": { productId: "15551234567", quantity: 2 },
  "an ordinary order": { productId: "MUG-01", quantity: 2 },
};

// The server's product catalog, standing in for a database lookup.
const catalog = new Map([
  ["MUG-01", { id: "MUG-01", name: "Mug" }],
  ["TEE-RED-M", { id: "TEE-RED-M", name: "Red T-shirt, M" }],
]);

function denylist(body: unknown) {
  const out = { ...(body as Record<string, unknown>) };
  delete out.password;
  delete out.phone;
  return out;
}

function nameAllowlist(body: unknown) {
  const b = body as Record<string, unknown>;
  return { productId: b.productId, quantity: b.quantity };
}

function loggableBody(body: unknown): Record<string, string | number> {
  const b = typeof body === "object" && body !== null ? (body as Record<string, unknown>) : {};
  const out: Record<string, string | number> = {};
  const product = typeof b.productId === "string" ? catalog.get(b.productId) : undefined;
  if (product) out.productId = product.id; // the catalog's ID, logged only when the lookup succeeds
  if (typeof b.quantity === "number" && Number.isInteger(b.quantity) && b.quantity >= 1 && b.quantity <= 99) {
    out.quantity = b.quantity;
  }
  return out;
}

console.log("logged url: ", url);
console.log("logged path:", url.split("?")[0]);
for (const [name, body] of Object.entries(bodies)) {
  console.log(name);
  console.log("  denylist: ", JSON.stringify(denylist(body)));
  console.log("  allowlist:", JSON.stringify(nameAllowlist(body)));
  console.log("  checked:  ", JSON.stringify(loggableBody(body)));
}

The denylist would strip a field called phone, but it passes mobile, the kind of field someone adds next month, and the number nested under productId. The allowlist of names drops mobile but passes the nested number, because it checks names and never looks at values. Both pass a phone number typed straight into productId. A format check such as /^[A-Z0-9-]{1,20}$/ passes it too, because eleven digits fit that pattern. loggableBody logs a product ID only when the catalog lookup finds it, and quantity only when it is a small integer, so neither phone number reaches the log and the ordinary order is still recorded.

The analytics call has the same shape of problem.

TypeScript
analytics.identify(user.id, { ...user });

Ask what the analytics service needs to do its job. It counts distinct users and groups them, and an opaque ID plus a plan name does both. Send { plan: user.plan } and nothing else. The model prompt gets the same treatment: the summary needs the conversation, so send only the excerpts it needs with unnecessary personal details removed, since the conversation can itself contain a phone number or an address, and leave the user record out. Check the provider's data-retention terms as well, because what you send may be stored.

The verification URL has two problems: it carries the phone number, and it carries the code. The text message does not need a link. It can contain the code, and the customer types it on the account page, where the server checks it against that customer's pending verification, expires it after a few minutes and limits the number of attempts. A six-digit code is safe in that setting because the session already says whose code it is, and the attempt limit keeps anyone from trying all million.

Top: an SMS link with the phone and code in the URL can reach access logs, browser history and, when the Referrer-Policy is loose, third parties via the Referer header. Bottom: an SMS code with no link goes to the account page, where the customer types it, then to a server check that expires the code and limits attempts. A session box, which says whose code it is, feeds the server check, and the server check reads that customer's pending verification record.
A link carrying the phone and code is recorded wherever the URL is; a code typed on the account page is checked against the pending verification of the customer the session names, with an expiry and an attempt limit.

If you want a link, put a token in it instead of the phone number. Generate the token with a cryptographically secure random generator, at 128 bits or more, and store its association with the pending verification on the server. Expire it, accept it once, and rate-limit attempts. The landing page exchanges the token for a session with restricted rights and redirects to a clean URL. A six-digit code cannot do this job alone, because a code used as the only lookup key can be guessed in a million tries.

Sensitive values in URL paths or query strings can reach access logs, browser history and third parties, so check which components record the URL. A URL can also reach third parties through the Referer header. Modern browsers send only the origin to other sites by default, but a page with a looser Referrer-Policy, or an older browser, sends the full URL with its query string.

Secrets

The SMS key in src/sms.ts is a restricted value in a file every developer and every CI job can read. Once it is committed, deleting it in the next commit leaves it in the repository's history, where anyone with a clone can find it. If the repository has ever been public, assume the key is stolen: hosting platforms and attackers both scan public commits for credentials.

A secret should live in a secrets manager or your deployment platform's secret store. The running service receives it at start-up, as an environment variable or a mounted file, and the code refuses to start without it.

TypeScript
const smsApiKey = process.env.SMS_API_KEY;
if (!smsApiKey) {
  throw new Error("SMS_API_KEY is not set");
}
A sequence: create new key, then, inside a box marked both keys valid, deploy with it and confirm that traffic uses the new key, then revoke old key. A separate key exposed node, revoke it promptly, has a dashed arrow straight to revoke old key; a note reads skips the sequence.
Scheduled rotation runs two keys side by side until traffic uses the new one; an exposed key skips the sequence and is revoked promptly.

Environment variables have leaks of their own. Child processes inherit them, and debug pages or crash reports that dump the environment copy them out. Diogo Mónica's essay on environment variables walks through these paths. Whichever way you inject the secret, keep it out of anything that prints configuration.

Rotation replaces an existing key. For scheduled rotation, run two keys side by side: create the new key, deploy the service with it, confirm that traffic is using it, then revoke the old one. Doing this on a schedule keeps the steps practiced. If a key is exposed, revoke it promptly instead. The side-by-side sequence leaves the leaked key valid while it runs, and restoring service afterward may involve some downtime. A pre-commit scanner such as gitleaks catches credentials that match its configured rules, so review what it finds and check for secrets the rules miss. That is cheaper than rotating after a leak.

Test data and screenshots

The twenty production rows in fixtures/users.json are real customers' names, emails and phone numbers, now copied into every laptop and CI runner that checks out the code. Test data should be generated. Use the example.com domain, which is reserved for documentation, and numbers ending in 555-0100 through 555-0199, which are reserved for fictional use in North America.

TypeScript
const fixtureUsers = Array.from({ length: 20 }, (_, i) => ({
  id: i + 1,
  email: `user${i + 1}@example.com`,
  phone: `+1202555${String(100 + i).padStart(4, "0")}`,
  displayName: `Test User ${i + 1}`,
}));

To spot a fixture built from real data, look for realistic personal email domains, names that look like real people, and IDs in the same format and range as production. Then ask where the file came from. Screenshots carry the same risk. A bug report, a design doc or a demo video captured from a real admin screen shows real customers, so capture them from a seeded demo account.

Before a field leaves your database, you should be able to name where it is going, why that destination needs it, who can read it there and when it will be deleted. If any answer is missing, withhold the field until the destination and its handling rules are understood. Apply the same review to an ID that can link records to a person.

Check your understanding

7 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1A sessions table stores token, a random 32-byte value the browser sends as a cookie to stay logged in. Which class fits token?
    Show the answer

    Answer C. Whoever holds the token is logged in as that user, which is account takeover, the harm that defines the restricted class. Internal fits a value like a row ID that stays harmless in a stranger's hands. A session token is harmless only until someone presents it.

  2. 2The growth team wants conversion rate per day and average cart total. An agent wrote this tracking call. What should it send instead?
    TypeScript
    analytics.track("checkout_completed", {
      userId: user.id,
      email: user.email,
      phone: user.phone,
      cartTotal: order.total,
    });
    Show the answer

    Answer B. Counting conversions and averaging totals needs an opaque ID and the amount. A SHA-256 of an email or phone number still identifies the person: anyone with a list of emails can hash it and match, and phone numbers are few enough to hash every one. A contract limits what the provider may do, but the copy still sits in their systems.

  3. 3An agent added a fallback so the app starts on a new laptop without any setup. What is wrong with it?
    TypeScript
    const smsKey = process.env.SMS_API_KEY ?? "sk_live_example";
    Show the answer

    Answer A. Every developer and CI runner that clones the repository now has the live key, and deleting the line later leaves it in history. Setting the variable in production changes nothing about that, since the leak is in the repository. The fallback also hides a missing variable; the code should refuse to start without it.

  4. 4A scanner finds a live payment API key in a commit pushed to a public repository three weeks ago. A later commit deleted it. What do you do first?
    Show the answer

    Answer D. Three weeks in a public commit means the key should be treated as stolen, because attackers scan public commits. Rewriting history or hiding the repository does nothing about copies already taken. Deploy-then-revoke is the right sequence for scheduled rotation, but here it keeps a stolen key valid until the deploy finishes.

  5. 5The app caches profiles for 24 hours, sends identify with the user's email to an analytics provider, logs request bodies for 90 days, and keeps nightly backups for 30 days. A customer asks to be deleted and this runs. An hour later, where can their email still be read?
    TypeScript
    async function deleteCustomer(id: number) {
      await db.query("DELETE FROM users WHERE id = $1", [id]);
    }
    Show the answer

    Answer C. Each copy was made when the data was written, so deleting the source row reaches none of them. A read-through cache still serves its stored entry until it expires, the provider keeps its own copy, the log line stays for 90 days, and each backup keeps the row until it ages out. Deletion has to reach every place a copy of the field lands.

  6. 6An agent built test fixtures by exporting 50 rows from the staging database, which is refreshed from a production snapshot every night. It says the data is safe because it never touched production. Is it?
    Show the answer

    Answer A. Staging that is refreshed from production holds production's customers, so the export copies real names, emails and phone numbers onto every laptop and CI runner that checks out the code. A private repository keeps out strangers, yet every developer and build can still read the rows. Generate fixtures with example.com addresses and numbers from 555-0100 to 555-0199.

  7. 7Customers get this receipt link by email. The site is served only over HTTPS, and its pages send a strict Referrer-Policy. Where can the email address in the link still end up?
    Output
    https://shop.example/receipt?email=alice@example.com&order=8812
    Show the answer

    Answer B. HTTPS hides the path and query from the network, but the origin server and any CDN or reverse proxy that terminates TLS see the full URL, and access logs often record it. The browser keeps it in history too. The strict referrer policy closes the path to other sites, so the Referer option no longer applies. Put an opaque token in the link, or no identifier at all.

You understand it when you can

Drill

An agent integrated error tracking by sending each exception's full context, including request headers and form fields, to a third-party service, and put password-reset tokens in the query string of the reset link. Find every place sensitive data now lands outside your control.

Topics

Data Classification
Sorting data into sensitivity classes so handling rules follow the class instead of case-by-case judgment.
Personal and Regulated Data
Personal information and health, financial and children's data, and the laws that govern them.
Secrets Management
Keeping credentials out of code and logs, in a secrets store, injected at run time and rotated on a schedule.
Data Minimization and Retention
Collecting less, keeping it for a shorter time, and deleting it on schedule.
Where Sensitive Data Leaks
Logs, URLs, error messages, analytics, crash reports, backups and the prompts sent to models.

Further reading

Start here

  • OWASP Secrets Management Cheat Sheet. Reference.

    The reference answer to how a secret is stored, injected into a running service, rotated and detected when it leaks into code or CI.

Short reads

Watch

Books

  • Data Privacy: A Runbook for Engineers. Nishant Bhajaria, 2022.

    An engineer's guide to classifying data by privacy risk, cataloging where it lives, controlling what is shared with third parties, and building deletion into the architecture.

  • Practical Data Privacy: Enhancing Privacy and Security in Data. Katharine Jarmul, 2023.

    Covers data governance, pseudonymization and anonymization, and building privacy into data pipelines, so you can minimize what sensitive data flows downstream.

Primary sources