Building Software
Sections

Contents Section 7, Operations

Shipping Change Safely

Version control, deployment pipelines, migrations, feature flags and rollback: the machinery for changing a live system many times a day.

Section 7 of 9, subsection 4 of 5. Depth: do. Core 18 of 25. A 10-minute read.

Mistakes to catch in review

  1. A migration that locks a large table, or drops a column the running version still reads.

  2. A large agent-generated change that mixes refactoring with behavior changes, so nobody can review it properly.

  3. A deployment with no rollback path, or a rollback that can't undo the data migration that shipped with it.

  4. A configuration change applied by hand in production and lost on the next deploy.

Two Versions at Once

An agent hands you a pull request titled "Store prices in cents." Forty files changed. The tests pass. Buried in the diff is one migration:

SQL
ALTER TABLE products ADD COLUMN price_cents integer;
UPDATE products SET price_cents = round(price * 100);
ALTER TABLE products ALTER COLUMN price_cents SET NOT NULL;
ALTER TABLE products DROP COLUMN price;

The application code now reads and writes price_cents everywhere. The agent also renamed ProductRepo to ProductStore while it was in there, moved six files, and changed how discounts round. On your laptop, with an empty database and one copy of the app, all of it works.

In production this change will break checkout for a few minutes, may freeze the products table while it runs, and cannot be undone by rolling back. None of that shows up in the tests, because the tests run one version of the code against a database that was created from scratch. Production runs two versions of the code against one database that already holds your data.

Old code and new code share one database

A deploy takes time. Suppose you run 20 app instances and your platform replaces them four at a time, waiting for each new batch to pass a health check before moving on. That is five batches, and if each takes a minute, then for about five minutes some requests are served by the old code and some by the new. This is a rolling deploy, and nearly every deployment method has a window like it.

The database has no such window. A migration changes the schema at one instant, for every instance at once.

A box labeled Migration, DROP COLUMN price, points down into a single database cylinder labeled one schema. On the left, a box labeled Old instances, read price, connects to the database with a dashed arrow labeled fails. Below it, a box labeled New instances, read price_cents, connects with a moving teal arrow labeled works. A note at the bottom reads: one schema, two versions.
Mid-rollout, old and new instances query the same database. If the migration drops price first, every request an old instance serves fails until that instance is replaced.

If the migration runs before the rollout, then when DROP COLUMN price commits, the 20 old instances are still running queries like SELECT id, name, price FROM products. Postgres answers each one with column "price" does not exist.

Every product page and every checkout served by an old instance fails until that instance is replaced. Even without the drop there would be a second break: old instances inserting a new product do not know about price_cents, so their inserts leave it null and violate the NOT NULL constraint.

If the migration runs after the rollout instead, the new instances start first and query price_cents, which does not exist yet. Either order breaks something. This model counts failed requests in each minute of the rollout, for both orders and for a schema that has both columns:

TypeScriptRuns in your browser
const INSTANCES = 20;
const BATCH = 4; // instances replaced per minute
const REQUESTS_PER_INSTANCE = 100; // per minute

type Version = "old" | "new";
const columnRead: Record<Version, string> = { old: "price", new: "price_cents" };

// Failed requests in each minute, from "nothing replaced yet" to "all replaced".
function rollingDeploy(columnsInSchema: string[]): number[] {
  const failedPerMinute: number[] = [];
  for (let replaced = 0; replaced <= INSTANCES; replaced += BATCH) {
    let failed = 0;
    for (let i = 0; i < INSTANCES; i++) {
      const version: Version = i < replaced ? "new" : "old";
      if (!columnsInSchema.includes(columnRead[version])) failed += REQUESTS_PER_INSTANCE;
    }
    failedPerMinute.push(failed);
  }
  return failedPerMinute;
}

console.log("migration first, only price_cents:", rollingDeploy(["price_cents"]));
console.log("migration last, only price:       ", rollingDeploy(["price"]));
console.log("expanded, both columns:           ", rollingDeploy(["price", "price_cents"]));

So at every moment during a deploy, the schema must work for both the version of the code that is leaving and the version that is arriving. That rule applies to every schema change. A migration that only works with the new code is safe only if you stop all traffic while you deploy.

The same rule covers anything else two versions share, such as messages in a queue or JSON stored in a cache. If new code writes a payload old code cannot read, old workers draining that queue during the rollout will fail on it.

What the migration does to the table

The second problem is locking. Most ALTER TABLE forms in Postgres take an ACCESS EXCLUSIVE lock, which blocks every other query on the table, reads included, until the transaction that took it commits.

Some of those statements finish in milliseconds. Adding a nullable column only updates the catalog. Since Postgres 11, adding a column with a constant default also avoids rewriting the table.

Other forms do real work while holding the lock. SET NOT NULL scans every row to check for nulls. Changing a column's type usually rewrites the whole table. A plain CREATE INDEX does not block reads, but it blocks inserts, updates and deletes until the index is built; CREATE INDEX CONCURRENTLY avoids that, and it cannot run inside a transaction.

Now look at the agent's file again. Many migration tools wrap each migration file in a single transaction on Postgres. If this one does, the ACCESS EXCLUSIVE lock taken by the first ALTER is held through the UPDATE, which touches every row. On a table with tens of millions of rows that update can run for minutes, and for all of those minutes nothing can read the products table.

A chain of waiting read from right to left. On the right, a container labeled New queries, reads included, holds three boxes: Read, Write, Read. An arrow labeled queued points from it to a highlighted box labeled ALTER TABLE, needs ACCESS EXCLUSIVE. An arrow labeled waits for points from the ALTER TABLE box to a box labeled Slow report, transaction open 30 s, which points down to a cylinder labeled products table. A note reads: millisecond migration, 30-second outage.
The ALTER TABLE cannot take its lock until the slow report's transaction finishes, and every new query on the table, reads included, queues behind the ALTER. A millisecond migration becomes a 30-second outage.

Even a fast statement can stall a table. To take its lock, the ALTER must wait for every transaction already using the table to finish. While it waits, every new query on that table queues behind it.

One slow report holding a transaction open for 30 seconds turns a millisecond migration into a 30-second outage. The defense is to set a short lock timeout in the migration, so it gives up and can be retried instead of blocking everyone:

SQL
SET lock_timeout = '2s';
ALTER TABLE products ADD COLUMN price_cents integer;

When you review a migration, ask of every statement what lock it takes and how much work it does while holding it. The PostgreSQL ALTER TABLE documentation lists the lock level for each form; other databases have their own rules.

Expand, migrate, contract

The fix for both problems is to split one incompatible change into several compatible ones, each deployed on its own. The pattern is usually called expand and contract, or parallel change. You first expand the schema so it supports both the old and the new shape, then move the code and the data across, then contract by removing the old shape once nothing uses it.

For the cents change, the sequence looks like this. Each step is a separate deploy, and each one finishes rolling out before the next begins.

  1. Add price_cents as a nullable column. Nothing reads or writes it yet, so old code is unaffected.
  2. Deploy code that writes both columns on every insert and update, and still reads price.
  3. Backfill existing rows in small batches, outside any long transaction.
  4. Deploy code that reads price_cents. It still writes both columns.
  5. Deploy code that writes only price_cents. If price is declared NOT NULL, drop that constraint first, or inserts that leave it out will fail.
  6. Once no running code touches price, add the NOT NULL constraint and drop price.

The backfill in step 3 walks the table one range of ids at a time, so each transaction holds its row locks briefly:

SQL
UPDATE products
SET price_cents = round(price * 100)
WHERE id >= $1 AND id < $1 + 5000;

A script runs that statement for each range up to the highest id, pausing between batches if the database is busy. It rewrites every row, including rows that already have a value. That matters because during step 2's rollout a new instance could write both columns and an old instance could then update only price, leaving the two out of step. Once step 2 is fully live, every writer keeps both columns in step, so one full pass after that point leaves the table consistent.

Three moments of one product row, left to right, each drawn as two cells, price and price_cents. First: a New instance box above writes both cells, with an arrow into each; the row is in step. Second: an Old instance box above updates only price, with one arrow into the price cell; the highlighted price_cents cell is stale, and the row is out of step, the two prices disagree. Third, after step 2 is live: a Backfill pass box reads price and rewrites price_cents with a moving teal arrow; the row is in step again. Arrows along the bottom run from in step to out of step to in step again.
During step 2's rollout, a new instance writes both columns and then an old instance updates only price, so the row's two prices disagree. A backfill pass that starts after step 2 is fully live rewrites every row and leaves the table consistent.

Check each step against the rule from the first section. During the rollout of step 2, old instances write only price and new ones write both, and the schema accepts both.

During step 4, some instances read price and some read price_cents, and both columns hold the same values because every writer still writes both. The column is only dropped after the code that read it is gone from every instance. A plain rename follows the same steps, with a copy in place of the conversion.

For the NOT NULL constraint on a large table, Postgres lets you avoid holding the heavy lock during the scan: add a CHECK (price_cents IS NOT NULL) NOT VALID constraint, run VALIDATE CONSTRAINT separately (it takes a lighter lock), and since Postgres 12, SET NOT NULL can then skip its own scan by using that validated check.

This takes six deploys instead of one, which is practical only when a deploy is cheap and automated.

What a rollback leaves behind

A rollback redeploys the previous version of the code. Migrations and every other piece of external state stay exactly as the bad release left them.

Try it on the agent's original migration. The new code has a bug, so you roll back. The old code starts, queries price, and finds no such column. The rollback has made things worse, because the only version that works with the current schema is the one you just removed.

You could write a down migration that re-adds price and recomputes it from price_cents. That works here only if every stored price is an exact number of cents and fits in the integer range:

SQLRuns on SQLite in your browser
CREATE TABLE products (id integer PRIMARY KEY, name text, price numeric);
INSERT INTO products VALUES (1, 'Mug', 12.50), (2, 'Poster', 19.999), (3, 'Sticker', 0.10);

-- The agent's conversion (the DROP comes after it).
ALTER TABLE products ADD COLUMN price_cents integer;
UPDATE products SET price_cents = round(price * 100);

-- A down migration would rebuild price from price_cents.
SELECT name, price AS original, price_cents, price_cents / 100.0 AS rebuilt,
       price = price_cents / 100.0 AS matches
FROM products;

The poster's 19.999 comes back as 20. If a dropped column held values that existed nowhere else, no down migration can bring them back.

The expand-and-contract sequence fixes most of this as a side effect. Through step 4, the previous step's code still works against the current schema, so you can roll back one step at a time. Step 5 breaks that: once the code stops writing price, the column goes stale, and a version that reads price would serve old prices. Keep the dual writes until you would no longer restore such a version. If you must restore one anyway, repair price from price_cents and verify it first. The final drop cannot be rolled back, and by then the code that needed the column has been gone for days.

Data is one kind of state a rollback leaves behind. Before you trust a rollback, list the others the release could have changed:

  • Feature flags live in a flag service or a config table, and redeploying code does not reset them.
  • Cache entries written by the new code stay in the cache, in whatever format the new code used, until they expire.
  • Jobs and messages the new code enqueued are still waiting to be processed, possibly by the old code.
  • Emails, webhooks and charges already sent to the outside world stay sent.

Feature flags are also a tool for making rollback cheaper. If step 4's read path sits behind a flag, you can deploy the code with the flag off, turn it on for a small fraction of users, and turn it off in seconds if something is wrong, without a deploy. This separates deploying code from releasing behavior. Every flag you add is also a branch in your code that has to be tested both ways and deleted when the rollout is done.

Canary deploys limit exposure at the instance level instead, sending a few percent of traffic to the new version first and watching its error rate before sending the rest.

Sometimes rollback is the wrong move. If the release included an irreversible migration, or the bug is small and the fix is one line, rolling forward with a new release can be safer than going back. Decide which one you would use before you ship.

Split the change so someone can review it

The migration is only one problem with the pull request. It also renames a class, moves six files, and changes how discounts round, all in one diff. A reviewer reading forty files of renames will not notice that one line in the discount code went from Math.floor to Math.round. That line changes what customers pay.

A refactor should change structure and leave behavior alone. Mixed with a behavior change, neither can be verified: the reviewer cannot confirm the refactor is behavior-neutral, and cannot find the behavior change in the noise.

Ask the agent to split the work into commits that each do one kind of thing:

  1. Rename ProductRepo to ProductStore and move the files. No other edits. The existing tests pass unchanged.
  2. Add price_cents and dual writes.
  3. Change discount rounding, with a test that shows the new result.

Each commit should build and pass tests on its own. The first can be reviewed by asking a single question: did anything other than names and paths change? The third is small enough to read line by line. If the rounding change turns out to be wrong, it can be reverted alone with git revert, without undoing the rename.

Two questions expose a mixed change. Could a reviewer confirm this commit changes no behavior, or name exactly the behavior it changes? If this commit were reverted by itself, what else would break? If the honest answer to the first is "not really," the commit needs splitting.

Configuration that lives only in production

The last way a change goes wrong has no diff at all. During an incident, someone raises the database pool size from 10 to 30 by editing an environment variable in the hosting console. The errors stop. Two weeks later a routine deploy applies the configuration from the repository, which still says 10, and the incident comes back with nobody remembering why it went away the first time.

Two rows. Top row: a box labeled Console edit, during the incident, points with a teal arrow labeled 10 to 30 to a highlighted box labeled Production, pool 30, errors stop. Bottom row: a box labeled Repository, still says 10, points with an arrow labeled deploy to a box labeled Production, pool 10, incident back. A dashed arrow, marked two weeks later, runs down from the top Production box to the bottom one. A note reads: commit what is running.
The console edit fixed production, but the repository still says 10. Two weeks later a routine deploy rebuilds production from the repository and the incident comes back.

Any setting that exists only in a live system will be overwritten by the next change that rebuilds it from source. The fix is to keep configuration, and the infrastructure definitions around it, in version control and change them through the same review and deploy path as code. When an emergency forces a hand edit, the follow-up is a commit that makes the repository match what is running.

You can check for drift directly: terraform plan reports resources that differ from the code, and kubectl diff shows how the cluster differs from a manifest. The question behind both is whether you could rebuild this environment from the repository right now and get what is running today.

Before any release goes out, you should be able to say, for each change in it, what state it touches, whether old and new code can both run against that state at the same time, and what you would do to undo it.

Check your understanding

7 questions on this lesson. Pick an answer to see whether it holds and why.

  1. 1An agent's migration runs this on a Postgres orders table with 80 million rows, during business hours. What happens to traffic while it runs?
    SQL
    CREATE INDEX idx_orders_customer ON orders (customer_id);
    Show the answer

    Answer C. A plain CREATE INDEX lets reads through and blocks writes for the whole build, which on 80 million rows can take many minutes of stalled checkouts. CREATE INDEX CONCURRENTLY avoids that, but it must be asked for, and it cannot run inside a transaction, which matters if the migration tool wraps each file in one. Most ALTER TABLE forms block reads as well; this statement lets them through.

  2. 2A reporting query has read orders inside a transaction that will stay open for another 40 seconds. This migration starts now, and new queries on orders keep arriving. What happens?
    SQL
    SET lock_timeout = '2s';
    ALTER TABLE orders ADD COLUMN gift_note text;
    Show the answer

    Answer C. Adding the column is quick once the lock is held, but the ACCESS EXCLUSIVE lock has to wait for the report's transaction to end, and new queries on orders queue behind the waiting ALTER. The lock timeout caps that queue at 2 seconds, after which the migration fails and can be retried. Without it, the table would stall for the full 40 seconds the report still has to run. The report is not cancelled; a waiting lock request never ends another transaction.

  3. 3During a rolling deploy, an old worker takes a job that a new instance enqueued. What happens?
    TypeScript
    // Old worker, still running on half the fleet:
    const job = JSON.parse(message) as { orderId: string; amount: number };
    await charge(job.orderId, job.amount);
    
    // New producer, deployed in this release:
    queue.send(JSON.stringify({ orderId: "ord_77120", amountCents: 4599 }));
    Show the answer

    Answer D. A TypeScript as cast only tells the compiler what to assume; at runtime the parsed object has amountCents and no amount. The two programs are compiled separately, so nothing compares their shapes. The new producer should send both fields until no old worker is left, or the workers should be updated to accept both before the producer changes.

  4. 4The tax-rounding change below shipped inside a commit titled "Rename InvoiceRepo to InvoiceStore" that touched 30 files. In the two weeks since, other commits added a dozen new callers of InvoiceStore. Now finance reports invoices a cent high. What does git revert of that commit do?
    Diff
    -export class InvoiceRepo {
    +export class InvoiceStore {
    @@
    -    const tax = Math.round(subtotal * rate);
    +    const tax = Math.ceil(subtotal * rate);
    Show the answer

    Answer B. A revert undoes everything the commit changed, so the rename comes out with the bug fix. The class goes back to InvoiceRepo, while the callers added since still refer to InvoiceStore, so the revert either conflicts where they touch the same lines or leaves code that no longer compiles. Had the rename and the rounding change been separate commits, the rounding commit could be reverted on its own.

  5. 5A release turned on a new checkout flow behind a flag, wrote cart entries to Redis in a new format with a 7-day TTL, and sent confirmation emails from a new template. You roll back the code. Which of these does the rollback undo?
    Show the answer

    Answer A. A rollback redeploys the previous code and nothing else. The flag keeps its value until someone changes it, the cart entries stay in the new format until they expire or are cleared, and the old code must be able to read them or it will fail on them. Emails that went out stay sent.

  6. 6Your pipeline applies deploy/config.yaml on every deploy. During an incident, someone raised the pool size to 30 in the hosting console. On Tuesday a deploy ships that changes only a CSS file. What is the pool size afterwards?
    YAML
    # deploy/config.yaml
    database:
      pool_size: 10
    Show the answer

    Answer B. The pipeline rebuilds the configuration from the repository on every deploy, whatever the code change was, so the hand edit is overwritten and the incident can return. The follow-up to any emergency console change is a commit that makes the file match what is running. The content of a deploy does not decide which settings it applies.

  7. 7This busy Postgres table has 50 million rows, and note is null in about 30 million of them. The migration sets a 2-second lock timeout. Which change finishes quickly without rewriting the table, scanning it or updating millions of rows?
    SQL
    CREATE TABLE events (
      id bigint PRIMARY KEY,
      amount integer NOT NULL,
      note text  -- nullable, and no CHECK constraint on it
    );
    Show the answer

    Answer D. Adding a nullable column with no default only changes the catalog, so once it gets its lock it finishes in milliseconds. Changing integer to bigint rewrites all 50 million rows while holding an exclusive lock, and the lock timeout does not help, because it limits only the wait for a lock, not how long the lock is held. With no CHECK constraint to rely on, SET NOT NULL scans every row under that same lock, and here it fails anyway, because the nulls are still there. The single UPDATE writes about 30 million rows in one long transaction; backfills go in small batches.

You understand it when you can

Drill

An agent wrote one migration that renames the email column to email_address and deployed it in the same release as the code that reads the new name. Find what breaks for requests served by old instances during the rollout and what a rollback leaves behind, then rewrite it as expand-and-contract steps.

Topics

Version Control Discipline
Small commits, readable history, branches and reverts as the record of why the code changed and the way to undo it.
Deployment Strategies
Blue-green, canary and progressive rollouts that limit how many users see a bad change.
Feature Flags
Separating deploying code from releasing features, and removing flags before they become permanent branches.
Expand-and-Contract Migrations
Changing schemas in steps that keep old and new code working at the same time.
Rollback and Roll-Forward
Knowing in advance how to undo each change, and when fixing forward is the safer move.
Infrastructure and Configuration as Code
Keeping environments and settings in version control so they are reviewed, repeatable and recoverable.

Further reading

Start here

Short reads

Watch

Books

  • Continuous Deployment: Enable Faster Feedback, Safer Releases, and More Reliable Software. Valentina Servile, 2024.

    A current, hands-on treatment of shipping to production on every commit, with feature flags, dark launches, expand-and-contract changes and small incremental commits.

  • Infrastructure as Code: Designing and Delivering Dynamic Systems for the Cloud Age. Kief Morris, 2025, 3rd edition.

    Explains why environments and configuration belong in version control and pipelines, and how to test and roll out infrastructure changes instead of editing production by hand.

  • Refactoring Databases: Evolutionary Database Design. Scott W. Ambler and Pramodkumar J. Sadalage, 2006.

    The classic catalog of schema refactorings, each done with a transition period in which old and new schemas coexist, which is where expand-and-contract migrations come from.

Primary sources

  • ParallelChange (Martin Fowler's bliki). Reference.

    Defines the expand, migrate, contract sequence for making a backward-incompatible change safely, with worked steps that apply directly to a column rename.

  • PostgreSQL Documentation: ALTER TABLE. Manual.

    Documents which ALTER TABLE forms take an ACCESS EXCLUSIVE lock or rewrite the table, the facts you check before a migration blocks production traffic.