Building Software
Sections

Contents Section 8, Security

Protecting Data in Transit and at Rest

How data moves securely between systems, and how it stays protected where it is stored.

Section 8 of 9, subsection 4 of 6. Depth: explain. A 10-minute read.

Mistakes to catch in review

  1. TLS certificate verification disabled in a client to get past an error.

  2. Home-made encryption, a hardcoded key or a reused initialization vector.

  3. An encryption key stored next to the data it protects.

  4. Database exports and backups left unencrypted in a publicly readable storage bucket.

Where the Key Lives: Encryption on the Wire and on Disk

An agent writes a small Node.js script that pulls daily exchange rates from a provider. The provider's page opens fine in your browser, but the script fails with Error: unable to verify the first certificate. The agent tries again, adds one option, and the error goes away:

TypeScript
https.get("https://rates.example/v1/latest", { headers: auth, rejectUnauthorized: false }, onRates);

The URL still starts with https. The connection remains encrypted, but rejectUnauthorized: false disables the certificate checks that establish whether the peer is the intended server.

Data is exposed in two places: while it moves between machines, and while it sits on one. In both places, a protection is worth exactly as much as the list of attackers it stops. Most encryption mistakes in agent-written code come from adding encryption without asking which attacker it is for, or where the key ended up.

What TLS does during the handshake

In a full TLS 1.3 handshake with server certificates and early data disabled, the client and server establish session keys and authenticate the server before the application request is sent.

Each side contributes random values, and they end up with shared session keys that someone copying every packet cannot compute. For authentication, the server sends a certificate, a document that says "this public key belongs to rates.example", signed by a certificate authority. The server then signs part of the handshake with the matching private key, which shows it holds that key.

Diagram. A certificate for rates.example fans out to three separate checks side by side. Chain: the signatures lead back to a certificate authority already in the client's trusted list. Name: the certificate name must match the host. Date: the current date must fall inside the validity period. The trusted-CA box, the name check and the date check all point to a highlighted box: the client trusts the certificate only if all three pass.
The client runs three separate checks and trusts the certificate only when all three pass: the signatures lead to an authority it already trusts, the name matches the host, and today falls inside the validity period.

The client checks the certificate before trusting it. The chain of signatures has to lead back to a certificate authority the client already trusts; your operating system or language runtime ships that list. The name in the certificate has to match the host the client meant to reach. And the current date has to fall inside the certificate's validity period.

After the handshake, TLS encrypts and authenticates application data with the session keys. Packet headers and some record metadata remain visible. Anyone on the path, such as a café Wi-Fi operator or a compromised router, sees ciphertext they cannot read and cannot alter without the connection failing.

RFC 8446, the TLS 1.3 specification, states what the protocol provides: the server is authenticated, and the data is kept confidential and protected against modification. TLS protects the channel between its endpoints. Its scope excludes endpoint storage and traffic metadata:

  • Once the server decrypts your request, what happens to it there is outside TLS.
  • An observer still sees which IP addresses are talking, when, and roughly how much data moves. In most connections the hostname is visible too, because the client sends it unencrypted so the server can pick the right certificate.
  • The server learns nothing about who the client is, unless the client also presents a certificate (mutual TLS) or sends a credential inside the encrypted channel.

What rejectUnauthorized: false gives away

Certificate validation is the step that ties the session keys to the right server. The key exchange alone only guarantees that you share keys with whoever answered. If the client never checks who answered, an attacker on the path can answer.

The attacker intercepts your connection and presents their own certificate. Your client, not checking, completes the handshake with them. The attacker opens a second TLS connection to the real provider and relays traffic between the two. Both connections are encrypted, and the attacker reads and can rewrite everything in between: the headers and body your client sends, and the response it gets back.

Turning off verification keeps the encryption and discards the authentication. The switch has a different name in each stack, and you should recognize every form of it in a diff:

TypeScript
new https.Agent({ rejectUnauthorized: false });            // Node.js, one agent
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";            // Node.js: disables verification by default for new TLS connections
fetch(url, { tls: { rejectUnauthorized: false } });        // Bun's fetch

Python's requests library calls it verify=False, Go calls it InsecureSkipVerify: true, and curl calls it -k.

The certificate error almost always has a real cause with a fix that keeps validation on. In the opening case the provider's server sends its certificate without the intermediate certificate that links it to a trusted authority. Browsers often fill in that missing link themselves, by downloading it or from a stored list of intermediates, and Node does not, so the fix is for the provider to serve the full chain. If the server uses a certificate from a private certificate authority, which is common for internal services, give the client that authority's certificate, through the ca option or the NODE_EXTRA_CA_CERTS environment variable. If the hostname does not match, connect using the name the certificate was issued for. A flag added to get past an error in one environment tends to reach production, because production runs the same code path that reads the flag.

Between your own services, mutual TLS has both sides present certificates. The server requests a client certificate, and the client proves it holds the matching private key, all within the same connection. The server then knows which service is calling as well as the client knowing which server answered.

Encryption at rest, and which attacker it stops

"Encrypted at rest" can mean several different mechanisms. Each one protects against an attacker who gets the stored bytes without getting the key, and each one keeps the key in a different place.

Mechanism What it encrypts Stops Does nothing against
Disk or volume encryption Every block on the drive Someone with the physical disk, a discarded drive, or a raw volume snapshot Anyone reading through the running system: the application, a database user, a SQL injection
Database file encryption The database's data files Someone who copies the files off the server Anyone who can run queries
Application or field-level encryption Chosen fields, encrypted by your code before they reach the database A leaked dump, a database administrator, an injection that reads the column Anyone who controls the application, since the application holds the key

The running system holds the key for disk and database encryption, so anyone who reaches data through the running system gets plaintext. Major cloud storage services now encrypt objects on disk by default, and a bucket made publicly readable still serves plaintext to the world. The storage service decrypts for every reader it considers authorized, and a public bucket authorizes everyone.

Database exports and backups deserve the same scrutiny as the database. An export is the whole dataset in one file, and it tends to land in places with weaker controls than the database: a storage bucket, a laptop, an attachment on a support ticket. Suppose a backup job writes orders.sql.enc and the raw key orders.key into the same bucket. Whoever can read the bucket reads both, so the encryption adds nothing against the people the bucket already trusts or mistakenly exposes. Encrypt exports with a key the storage location cannot reach, and read the bucket's access policy as carefully as you read database grants.

A worked example: the key beside the lock

An agent adds field-level encryption for customer bank account numbers. To avoid hardcoding a key, it stores the key in a settings table, in the same database as the accounts table:

TypeScript
const NONCE = new Uint8Array(12); // twelve zero bytes

async function encryptAccount(db: Db, accountNumber: string): Promise<Uint8Array> {
  const row = await db.queryOne("SELECT value FROM settings WHERE name = 'field_key'");
  const key = await crypto.subtle.importKey("raw", Buffer.from(row.value, "hex"), "AES-GCM", false, ["encrypt"]);
  const sealed = await crypto.subtle.encrypt({ name: "AES-GCM", iv: NONCE }, key, new TextEncoder().encode(accountNumber));
  return new Uint8Array(sealed);
}

The code uses a well-reviewed library and a standard algorithm. It stores the encryption key in the same database as the encrypted data, and it reuses one nonce for every account number.

Before and after diagram. Left, key in the same database: one database holds the accounts table with ciphertext and the settings table with field_key. A database reader, meaning a dump, backup, read replica or SQL injection, has arrows into both tables, so it reads the ciphertext and the key. Right, key outside the database: the database holds only the accounts table with ciphertext. A key service, out of the database’s reach, supplies the key to the application at runtime. The same database reader has one arrow, into the accounts table, and reads ciphertext only.
With field_key in the settings table, anything that can read accounts can read the key as well. Moved to a key management service, the key is out of reach, and the same reader gets ciphertext with nothing to open it.

The first problem is where the key lives. A dump, a backup, a read replica or a SQL injection that reads the accounts table can read the settings table too. Encryption protects data from people who get the ciphertext without the key, and this design hands both to the same group: anyone who can read the database. The key has to come from a place the database and its backups cannot reach, such as a key management service or a secrets manager that injects it at runtime.

The second problem is the fixed nonce. AES-GCM runs AES in counter mode, which turns the key and nonce into a stream of pseudo-random bytes and XORs that stream with the plaintext. The same key and nonce always produce the same stream. XOR two ciphertexts made that way and the stream cancels out, leaving the XOR of the two plaintexts. The plaintexts below share the five-byte prefix acct=, so XORing their encrypted prefixes gives five zero bytes. The block then plays an attacker who knows one plaintext and its ciphertext, such as their own account number, and repeats the attack against fresh random nonces:

TypeScriptRuns in your browser
const enc = new TextEncoder();
const key = await crypto.subtle.generateKey({ name: "AES-GCM", length: 256 }, false, ["encrypt"]);
const seal = async (iv: Uint8Array, text: string) =>
  new Uint8Array(await crypto.subtle.encrypt({ name: "AES-GCM", iv }, key, enc.encode(text)));

const fixed = new Uint8Array(12);
const c1 = await seal(fixed, "acct=12345678");
const c2 = await seal(fixed, "acct=87654321");
console.log([...c1.slice(0, 5)].map((b, i) => b ^ c2[i])); // [ 0, 0, 0, 0, 0 ]

// The attacker XORs a plaintext they know with its ciphertext to get the keystream,
// then XORs the keystream with someone else's ciphertext.
const known = enc.encode("acct=12345678");
const recover = (mine: Uint8Array, theirs: Uint8Array) =>
  new TextDecoder().decode(theirs.slice(0, known.length).map((b, i) => b ^ known[i] ^ mine[i]));
console.log(recover(c1, c2)); // acct=87654321

const c3 = await seal(crypto.getRandomValues(new Uint8Array(12)), "acct=12345678");
const c4 = await seal(crypto.getRandomValues(new Uint8Array(12)), "acct=87654321");
console.log(recover(c3, c4) === "acct=87654321"); // false

Recovering the keystream works over the length of the known plaintext, so here it reveals every account number of the same length. Nonce reuse in GCM also leaks the internal key used for the authentication tag, which lets the attacker forge ciphertexts that decrypt without error.

The corrected version generates a fresh random nonce for every encryption and stores it in front of the ciphertext, so each encryption under a key uses a distinct nonce and the decryptor can find it.

TypeScriptRuns in your browser
const enc = new TextEncoder();

async function encryptField(key: CryptoKey, plaintext: string, rowId: string): Promise<Uint8Array> {
  const nonce = crypto.getRandomValues(new Uint8Array(12));
  const sealed = await crypto.subtle.encrypt(
    { name: "AES-GCM", iv: nonce, additionalData: enc.encode(rowId) },
    key,
    enc.encode(plaintext),
  );
  const blob = new Uint8Array(12 + sealed.byteLength);
  blob.set(nonce);
  blob.set(new Uint8Array(sealed), 12);
  return blob;
}

async function decryptField(key: CryptoKey, blob: Uint8Array, rowId: string): Promise<string> {
  const plain = await crypto.subtle.decrypt(
    { name: "AES-GCM", iv: blob.slice(0, 12), additionalData: enc.encode(rowId) },
    key,
    blob.slice(12),
  );
  return new TextDecoder().decode(plain);
}

const key = await crypto.subtle.generateKey({ name: "AES-GCM", length: 256 }, false, ["encrypt", "decrypt"]);
const blob = await encryptField(key, "12345678", "account:17");
console.log(await decryptField(key, blob, "account:17")); // 12345678

// The same blob pasted into another customer's row:
try {
  await decryptField(key, blob, "account:18");
} catch (e) {
  console.log("rejected:", (e as Error).name); // rejected: OperationError
}

The row ID goes in as associated data. It is authenticated without being encrypted. On decryption, supply the expected row ID from trusted record context, and a ciphertext copied from one customer's row to another's then fails to decrypt. With random 96-bit nonces, NIST's guidance for GCM (SP 800-38D) caps a single key at 2^32 encryptions, which tells you when a key has to be retired.

Home-made cryptography looks different from this. Watch for XOR against a password, base64 described as encryption, a hash of the key used as a keystream, AES in ECB mode, or AES-CBC with no authentication tag. The question that exposes all of them is: which library call performs the encryption, and does it produce a tag that is checked on decryption? If the answer is a loop the agent wrote, reject it. Use the high-level interface of a vetted library, such as AES-GCM or ChaCha20-Poly1305 from a mainstream library, libsodium's secretbox, or Google's Tink.

Envelope encryption and rotation

Once the key is out of the database, you still need a design for where it lives and how you replace it. Envelope encryption is the standard answer.

Each tenant, file or record gets its own random data encryption key (DEK). Your code encrypts the data locally with the DEK. It then asks a key management service to encrypt the DEK under a key encryption key (KEK) that never leaves the service. You store the wrapped DEK next to the ciphertext. On its own, the wrapped DEK is useless; unwrapping it needs a call to the service, which checks permissions and logs every request.

TypeScript
async function encryptRecord(kms: KmsClient, kekId: string, plaintext: string, rowId: string) {
  const dek = await crypto.subtle.generateKey({ name: "AES-GCM", length: 256 }, true, ["encrypt", "decrypt"]);
  const rawDek = new Uint8Array(await crypto.subtle.exportKey("raw", dek));
  return {
    kekId,
    wrappedDek: await kms.encrypt(kekId, rawDek), // kms: your key service's client
    blob: await encryptField(dek, plaintext, rowId),
  };
}

Rotation is where this design pays off. Say you have 40,000 tenants, one DEK each, and 3 TB of encrypted data. Rotating the KEK means unwrapping 40,000 DEKs with the old KEK and wrapping them with the new one. If each wrapped-DEK blob is 250 bytes, replacing 40,000 blobs writes about 10 MB; the 3 TB of data is unchanged. With one key encrypting everything directly, rotation means reading, decrypting and rewriting all 3 TB while both keys stay live.

Diagram of envelope encryption. A DEK, one per tenant, encrypts data locally into ciphertext, 3 TB, held in storage. The DEK is also sent into the key service, where the KEK, which never leaves the service, wraps it. Wrapped DEKs, 40,000 at 250 bytes each, come out of the key service into storage beside the ciphertext. For rotation, the stored wrapped DEKs go back into the key service, the new KEK rewraps them inside it, and the rewrapped DEKs return to storage, about 10 MB; the ciphertext is not rewritten.
Each DEK encrypts data locally; the KEK stays inside the key service, the DEK is wrapped there, and the wrapped DEK is stored beside the ciphertext. Rotating the KEK sends the 40,000 small wrapped DEKs back to the service to be rewrapped, about 10 MB, while the 3 TB of ciphertext stays as it is.

Rewrapping supports routine KEK rotation. If an attacker obtained the old KEK and the old wrapped DEKs, treat those DEKs as compromised: replace them and re-encrypt the data. The same holds for a DEK that leaked by itself, for example printed into a log, because rewrapping changes nothing for that tenant. Envelope encryption keeps that job to the affected tenant's data. Note also where the power moved: whoever holds permission to call decrypt on the KEK can read everything, so that permission is now the access control that matters.

Hashing, encryption or signing

Take password storage first. Login verification compares a submitted password against a stored password hash. Use Argon2id, scrypt or bcrypt, each with a per-user salt and deliberately slow. A fast hash like plain SHA-256 lets an attacker with a stolen table test billions of guesses per second on one GPU. Encryption is the wrong tool here, because whoever gets the key gets every password at once.

A different need is storing a customer's credential for a third-party service so your system can call it on their behalf. You need the original value back, so use authenticated encryption, as in the example above. A hash cannot be reversed, so it cannot serve this need.

Three rows, each a need pointing to a tool. Checking a password at login points to a slow, salted hash such as Argon2id, scrypt or bcrypt, which cannot be reversed. Using a stored credential, where you need the original value back, points to authenticated encryption such as AES-GCM, which the key reverses. Proving a file came from you and was not altered points to a digital signature such as Ed25519, made with the private key and verifiable by anyone with the public key.
Each need points at a different tool: a slow hash for passwords you only compare, authenticated encryption for secrets you must read back, and a signature for proving where a file came from.

A third need is letting recipients confirm that a file or message came from you and was not altered. A digital signature does that: sign with a private key (Ed25519 is a common choice), and recipients verify with the public key, which they must have obtained through a channel they trust. Anyone can verify; only you can sign. When both sides already share a secret, as with webhook secrets issued per customer, an HMAC does a similar job more cheaply, with the difference that anyone who can verify an HMAC can also create one. A bare SHA-256 checksum proves integrity only when the checksum reaches you through a channel the attacker cannot touch, since an attacker who can swap the file can swap its hash too.

For every piece of encryption an agent adds, write down where the key lives and who can read that place. The encryption protects the data only from people outside that group.

Check your understanding

7 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1Your service calls a payments API over HTTPS with certificate verification on. Someone records every packet on the network between you and the API. What can they learn?
    Show the answer

    Answer D. TLS encrypts application data, headers and body included, so the amounts and the key stay hidden. It leaves traffic metadata visible: IP addresses, timing, sizes, and in most connections the hostname, which the client sends unencrypted so the server can choose a certificate.

  2. 2The agent added the flag after this call failed certificate validation. Which change keeps the call working with full verification?
    TypeScript
    // inventory.internal uses a certificate issued by the company's private CA.
    const agent = new https.Agent({ rejectUnauthorized: false });
    https.get("https://inventory.internal/v2/stock", { agent }, onStock);
    Show the answer

    Answer B. Validation failed because the client did not know the authority that signed the certificate. Giving it that authority's certificate lets every check pass. Gating the flag on NODE_ENV looks safe, but the code that reads the flag ships to production, and one host with the wrong variable runs unverified. An internal network is also where an attacker who already got in sits.

  3. 3You publish a nightly file of exchange rates. Forty partners download it from a mirror you do not control, and each must be able to confirm the file came from you unchanged. What do you use?
    Show the answer

    Answer C. Only your private key can sign, and anyone with your public key can verify, which is exactly the need. A checksum on the same mirror can be swapped along with the file. With one shared HMAC secret, any of the forty partners can compute a valid tag for a forged file, and the same holds for a shared AES key.

  4. 4An agent says this function encrypts each value with a random nonce. What is wrong?
    TypeScript
    const iv = crypto.getRandomValues(new Uint8Array(12));
    
    async function seal(key: CryptoKey, text: string) {
      const ct = await crypto.subtle.encrypt({ name: "AES-GCM", iv }, key, new TextEncoder().encode(text));
      return { iv, ct };
    }
    Show the answer

    Answer A. The random value is drawn once, when the module loads, and every call reuses it under the same key. Two ciphertexts made with the same key and nonce XOR to the XOR of their plaintexts, and GCM's authentication key leaks too. Storing the nonce beside the ciphertext is normal; the nonce only has to be unique per key.

  5. 5You use envelope encryption with one data key per tenant, each wrapped by a key in your KMS. A bug printed one tenant's unwrapped data key into a log that many engineers can read. What is the smallest change that closes the exposure?
    Show the answer

    Answer D. The leaked value is the key that decrypts that tenant's data directly, so anyone who read the log can decrypt it without asking the KMS. Rewrapping changes only the wrapper around that same key. Re-encrypting every tenant also closes the exposure, but the other tenants' keys never appeared in the log, so that work buys nothing.

  6. 6A storage bucket of customer exports has the provider's default server-side encryption turned on. A policy change makes the bucket publicly readable. What does a stranger who downloads a file get?
    Show the answer

    Answer B. Server-side encryption protects the disks underneath. The storage service holds the key and decrypts for any request its policy allows, and a public bucket allows everyone. To protect an export from people who can read the bucket, encrypt it with a key the bucket cannot reach.

  7. 7An agent wrote this to protect users' saved notes. Which replacement fits?
    TypeScript
    function encrypt(text: string, password: string): string {
      const out = [...text].map((ch, i) => ch.charCodeAt(0) ^ password.charCodeAt(i % password.length));
      return btoa(String.fromCharCode(...out));
    }
    Show the answer

    Answer B. Repeating-key XOR has no authentication tag and leaks patterns: XOR two notes and the key cancels out, and any known stretch of text reveals the key. A longer password or a hashed one still gives a repeating key, the same broken construction. Authenticated encryption from a vetted library hides the text and makes a tampered note fail to decrypt.

You understand it when you can

Drill

An agent wrote a client for a partner API with certificate verification turned off 'for local testing', and encrypts stored API keys with a key defined as a constant in the same file. Find what an attacker on the network and an attacker who reads the repository can each do.

Topics

Encryption in Transit
TLS on every connection, certificate validation, and mutual TLS between services.
Encryption at Rest
Disk, database and field-level encryption, and which threats each one actually addresses.
Key Management
Where keys live, who can use them, envelope encryption and rotation.
Hashing, Signing and MACs
Tools for integrity and authenticity, and when each one is the right choice.
Using Vetted Cryptography
Relying on well-reviewed libraries, algorithms and modes instead of designing cryptography yourself.

Further reading

Start here

  • TLS Handshake Explained - Computerphile. Mike Pound, 2020. 17-minute explainer.

    Walks through how client and server agree keys and how the certificate proves the server's identity, which shows what you give up by disabling certificate verification.

Short reads

Watch

  • Practical cryptography with Tink - Neil Madden - NDC Security 2025. Neil Madden, 2025. 43-minute talk.

    Shows how to use Google's misuse-resistant Tink library, which chooses algorithms and nonces for you and manages keys as rotatable keysets, instead of assembling primitives by hand.

  • Crypto 101. Laurens Van Houtven, 2013. 46-minute talk.

    PyCon 2013 introduction that builds from block and stream ciphers to hashes, MACs and key exchange, showing how naive constructions break and why hashing, MACs and encryption each have a separate job.

Books

  • Real-World Cryptography. David Wong, 2021.

    A practitioner's map of which primitive to use for which job (authenticated encryption, MACs, signatures, key exchange, TLS), written for engineers rather than mathematicians.

  • Serious Cryptography: A Practical Introduction to Modern Encryption. Jean-Philippe Aumasson, 2024, 2nd edition.

    Explains authenticated encryption, hash functions, randomness and TLS's strengths and limits in enough depth that you can spot a reused IV or a home-made scheme.

  • Bulletproof TLS and PKI: Understanding and Deploying SSL/TLS and PKI to Secure Servers and Web Applications. Ivan Ristić, 2022, 2nd edition.

    The deployment reference for TLS 1.3, certificates, validation and PKI failures, from the author of SSL Labs.

Primary sources