The Bad Update Reached Every Replica. What Can We Restore?

A faulty importer sets north’s stock to zero. Replication does its job and sends that change to the other copies. Botie can reach the service, but every reachable view now contains the mistake.

We need history from before the damage, not merely another current copy. A backup snapshot and retained changes can reconstruct a chosen state if their recovery contract covers the required interval.

Check the replay interval, apply its replacements to a snapshot copy, then label the recovered state with its target position.

{
  "seq": 12,
  "state": {
    "north": 5,
    "south": 5
  }
}

Replicas can help keep a service running after a node fails. They commonly receive validly committed mistakes as well as intended writes. A backup protects a different need: recovering a previous state or a separate durable copy under its retention policy.

A backup on the same compromised account or storage boundary may share the incident’s failure mode. Choose isolation, retention and deletion controls according to the failures the maintenance service must recover from.

A healthy replica is useful, but it is not automatically a historical backup.

The recovery point objective describes acceptable data loss, often expressed as how far back the restored state may be from the incident. The recovery time objective describes how long the service can remain unavailable or degraded while recovering. These are requirements, not outcomes established by enabling a setting.

Measure both in restore drills. Include locating the backup, obtaining authorised keys, restoring data, checking application configuration and making the service usable. A quick file copy can be a small part of the complete recovery time.

The exercise’s target sequence illustrates a chosen point, not a measured RPO or RTO.

Our teaching snapshot contains seq and a dictionary of inventory balances. Each log record gives the next consecutive sequence and a full replacement for one balance. To restore a target, select positions after the snapshot up to that target and require every intervening position.

A missing event stops the operation. Publishing the remaining records would create a plausible image whose history is incomplete. Real recovery logs may have gaps in numeric positions, so their completeness checks use the database’s documented mechanism rather than this invented consecutive-integer rule.

The local function is a deterministic restore calculation, not a database backup tool.

snapshot_position = 10
target = 12
required = list(range(snapshot_position + 1, target + 1))
print(required)
print(13 in required)

[11, 12]
False

Copy the snapshot before applying changes so a failed attempt does not destroy the starting point. Replaying full replacements in sequence gives the state at the target under this fixture’s contract. A new key can appear in the replay even if the snapshot did not contain it.

After recovery, compare expected identities and balances and run application-level checks. Restoring bytes is not proof that indexes, access policies, credentials and application configuration are ready. Some services restore into a new resource that needs its settings reapplied.

DynamoDB documents restoration into a new table and lists settings that need separate attention.

Backups can contain the same sensitive tenant data as the live service. Restrict who can list, read, restore or delete them. Use the supported encryption mechanism and retain access to its keys for the entire recovery period. A perfectly intact encrypted backup is unusable if the recovery team cannot obtain the required key permissions.

Use separate development restore destinations and appropriately scoped roles during drills. Verify that a restored copy does not accidentally expose production data through a broadly accessible test service.

Recovery permissions should allow the planned procedure without becoming ordinary web-application privileges.

Restoring before the faulty import can also remove legitimate changes that happened afterward. Identify which later operations can be safely replayed, reconciled or re-entered. Keep evidence of external actions, such as deliveries already dispatched, because moving the database backwards does not undo them.

Freeze or route writes during cutover according to a documented plan. Otherwise the old and restored services can both accept new stock changes and diverge while the team believes recovery is complete.

Choosing a point is a business recovery decision as well as a storage operation.

Run a restore drill with representative size and application checks. Record which backup was used, its recoverable interval, the measured elapsed time and any settings that were missing. Test a missing-history case so the process refuses an unsupported target.

Repeat after material schema, encryption or infrastructure changes. A recovery document that nobody has exercised can preserve an obsolete assumption for months.

The workshop’s later target includes the faulty change deliberately. Inspect the target before treating a successful replay as the desired repair.