Skip to main content

Untested Backups Are Not Backups

· 2 min read
Sam Cookes
Sam Cookes
Senior Dev at cookedup.sh

We had point-in-time recovery enabled on every DynamoDB table. We had runbooks documenting the restore procedure. We had never actually tested any of it.

The Runbooks​

Seven operations runbooks covering the scenarios that keep you up at night: API key compromise (single client and at scale), Lambda rollback, session revocation for compromised accounts, database recovery, secret rotation, and emergency CDN invalidation.

Each one is step-by-step with the exact CLI commands. No ambiguity, no "consult the docs." When you're responding to a P0 at 2am, you want a recipe, not a reference.

The Verification​

Writing runbooks is documentation. Testing them is operations.

The backup verification script does four things for each DynamoDB table:

  1. Confirms PITR is enabled (fails fast if not)
  2. Restores to a temporary test table using the latest restorable point
  3. Compares record counts and samples records between source and restored tables
  4. Cleans up the test tables on exit (trap handler, so cleanup happens even on failure)
bash tool/bin/verify-backups.sh --profile my-profile

The whole run takes about 10 minutes — most of that is waiting for DynamoDB to create the restored tables.

What We Found​

Everything passed. Eight checks, zero failures. PITR enabled on both tables, restore completed successfully, record counts matched, sample records identical.

The first run actually failed — I used an AWS CLI v2 flag (--use-latest-restorable-point) on a machine running v1. The fix was trivial (--restore-date-time with the latest restorable timestamp), but it's exactly the kind of thing you don't want to discover during an actual incident.

The pre-flight checklist now includes "check tool versions before writing scripts."

The Neon Side​

Neon (our Postgres host) manages PITR through their console with branch-based recovery. You create a branch at a point in time, verify the data, then either promote it or delete it. The script documents the manual steps since Neon's API isn't scriptable the same way.

Why This Matters​

The gap between "we have backups" and "we can restore from backups" is where incidents become disasters. The verification script runs in CI — it can be scheduled weekly, monthly, whatever cadence matches your risk tolerance.

Untested backups are not backups. Now ours are tested.