“We can roll it back” is a comforting sentence. It becomes useful only when someone can explain what the previous release will encounter after it returns.

The old binary may still be available. The container registry may still have its digest. Your deployment tool may know exactly how to put that artifact back. None of those facts establishes that the old application can operate against the database, messages, configuration, and external effects left by the new one.

A rollback plan needs a compatibility check. The question is whether the selected recovery release can read and write the state that will exist when you need it. That is a release-engineering problem worth solving before the deployment starts.

Name the recovery target and the state it inherits

Start with an exact artifact, not “the last good version.” Record the image digest or release identifier, the configuration it needs, and the point in the migration sequence where returning to it is still supported.

Then inventory the state that survives an application replacement:

• Database columns, constraints, and stored values.
• Messages already published and jobs waiting in queues.
• Files, object-storage records, and cached serialized values.
• Feature flags, configuration keys, credentials, and permissions.
• Requests already accepted by external systems.

Some changes are compatible; some require a bridge release; some close the rollback window. A code revert does not cancel an email, reverse a payment, or restore a deleted field. Treat those effects as separate recovery work with explicit ownership.

The Amazon Builders’ Library discussion of rollback safety explains why old and new software must tolerate shared state and communication during a deployment. Its prepare/activate approach establishes reader compatibility before enabling new writes. It also warns that an activation can make the original pre-prepare version an unsafe rollback target.

The practical consequence: your recovery target can advance before your feature is fully enabled. Keep the compatible bridge available instead of assuming you can jump back to any historical release.

Build a small compatibility matrix

A matrix turns a vague promise into questions someone can test. For an illustrative change from an old status field to a replacement, write down the following:

R0 is the original release, R1 the compatibility bridge, and R2 the new reader. R1 writes both columns, so it requires expansion even while reading the old field. Wait means R2 requires all writers to be upgraded and both fields reconciled before its reads are safe. Yes* requires the old representation to remain current and retain its original semantics.

A Yes must mean correct behavior, not merely a process that starts. An old reader that silently misclassifies an order is worse than a loud schema error.

The matrix depends on the implementation. R0 support in the third row also requires the old representation to express every permitted value. If R2 introduces a state that R0 cannot represent, that row is unsupported. Change the rollout or narrow the supported recovery target; do not leave an optimistic label in the release notes.

For services communicating through queues, add equivalent rows for old and new message versions. Include delayed consumers and redelivery. The first successful request after deployment tells you little about a worker that wakes up tomorrow.

An expand/contract example with an honest rollback window

Suppose an application stores an order's state in status and wants to use fulfillment_state. For this example, the values have identical meanings. This is a column transition, not an excuse to smuggle in a new state machine.

The expansion might begin with:

-- Expansion only; PostgreSQL.
ALTER TABLE orders
    ADD COLUMN fulfillment_state text;

PostgreSQL's ALTER TABLE documentation documents both column operations and their locking behavior. Adding a nullable column is not permission to ignore lock acquisition, concurrent workload, or your migration tooling. Set appropriate operational limits and rehearse against representative conditions.

Expand and keep the old field authoritative

Deploy R1 so it reads the existing status and writes both fields within the same database transaction. R0 instances can continue to run while R1 rolls out, because status remains authoritative.

Backfill the new field in bounded batches. While R0 writers exist, they can update only status; therefore a completed first pass is not proof that the new column is current. Keep readers on status until all relevant writers are upgraded and a reconciliation pass verifies matching values.

This includes API servers, scheduled jobs, maintenance scripts, and consumers in other services. A deployment dashboard showing green for one service does not inventory every writer.

Switch reads without abandoning the old representation

After verifying writers and reconciliation, deploy R2 to read fulfillment_state while continuing transactional writes to both fields. Avoid introducing values that the old release cannot understand during the supported rollback window.

Returning to R1 is now straightforward in principle: the old field is still current. Returning to R0 needs another check. R0 will stop maintaining the new field, so roll forward again through a reconciliation phase before trusting it. A rollback can preserve customer service while invalidating a later migration precondition.

A fallback such as “use the old field when the new one is null” does not solve stale non-null values. Specify authority and synchronization instead of hoping a fallback expression hides the race.

Contract only after retiring the old recovery target

Removing status, dropping synchronization, or permitting new-only values is a separate release decision. It can close the R0/R1 rollback window.

Before contracting, establish a newer tested recovery artifact, verify that old readers and writers are gone, and decide what happens to delayed work. Do not automatically run a destructive down migration when replacing application code. Dropping the replacement column could discard information written after the original migration.

Expansion costs extra code and temporary duplication. That cost buys a controlled transition. Give the bridge an owner and a removal condition so temporary compatibility does not become permanent archaeological infrastructure.

Configuration and messages deserve the same scrutiny

A schema migration is easy to notice in review. A renamed environment variable or changed message meaning can be just as consequential.

Consider a worker that previously accepted queued and done but now emits cancelled. Keeping the JSON structure unchanged does not make that value compatible. Test how the old consumer handles it, including retries, dead-letter behavior, and any default branch that treats unknown values as success.

For configuration, keep the old release's required keys and access paths available during the rollback window. If a credential has been revoked, returning the binary does not return its permissions. Conversely, retaining a credential indefinitely creates its own maintenance problem. Record the time or condition for retiring it alongside the release target.

Decide Which Dependencies to Update Automatically addresses permission to merge an update. Compatibility answers a different question: what recovery remains possible once the update has changed running state?

Rehearse the transition, not just the old build

The useful rehearsal retains state across the whole sequence. Starting an old binary against a fresh old database proves the wrong thing.

1. Create representative old state in an isolated environment.
2. Run the supported expansion and deploy a mixture of old and bridge instances.
3. Exercise reads, writes, background jobs, retries, and concurrent updates.
4. Complete the transition and create state using the new release.
5. Return to the exact supported recovery artifact without resetting the database or queues.
6. Verify business results, message handling, and write correctness.
7. Rehearse rolling forward again, including reconciliation after old-only writers return.

Use fixtures for boundary values and representative workload behavior. Production-shaped testing does not require copying sensitive production data. Record what was exercised and what remains untested; do not describe a quiet smoke test as a complete recovery rehearsal.

Keep the command path reproducible. Making Local CI Commands Boring Enough for Humans and AI Agents explains why supported verification should be discoverable and repeatable. A recovery test nobody can rerun is weak evidence for next month's release.

Choose rollback, restore, or a forward fix explicitly

Rollback replaces application behavior. Restore replaces state from a recovery source. A forward fix introduces another change. They have different risks and can be combined, but they are not interchangeable verbs.

Restoring a database can lose or require reconciliation of valid writes accepted after the restore point. It also does not necessarily undo side effects in external systems. A forward fix may preserve more state but requires confidence in another artifact during an incident. A rollback is attractive when the previous release remains compatible and can reduce harm quickly.

Before release, record the supported target, compatibility evidence, window-closing changes, decision owner, and success signals. If the check fails during an incident, stop treating rollback as the default and use the documented alternative. Explain uncertainty to the incident owner before initiating an action that changes or discards state.

Make recovery a property of the release

A useful rollback plan names a version, describes the state it inherits, and demonstrates that the two work together. It also tells the operator when that claim stops being true.

Keep the matrix small, rehearse the actual transition, and retire recovery targets deliberately. The release is ready when the team can explain both how to move forward and which safe route remains if the result is wrong.

More at Slaptijack.