Test a restore in isolation

A backup is unproven until you restore it. Listing archive files proves only that an archive can be listed.

Use an isolated host, directory, database, or account. Verify checksums, ownership, permissions, service startup, representative records, and a user-visible workflow. Record elapsed recovery time.

Why isolation matters

Restoring over production is how you turn a backup test into an outage. One wrong flag, one stale path, and you overwrite the only good copy with a partial restore. Isolation gives you room to fail safely.

Pick a target that shares nothing with production: a separate directory on another filesystem, a throwaway VM, a database named shopdb_restore_test. Production keeps running while you learn whether the backup actually works.

What to verify

File restores need more than a byte count. Compare trees with diff -qr or rsync --dry-run. Check that symlinks point where you expect, that private keys still have mode 600, and that upload directories are writable by the application user.

Database restores need integrity checks after load. Run pg_restore into a separate database name, then query row counts and the newest timestamp against what the backup job logged. Start the application against the restored database only if the data checks out.

Time the whole workflow from “decision to restore” to “user-visible action works”. That measured duration is your real RTO, not the number you wrote in a doc six months ago.

Restore to a new directory, never over the live data first:

sudo mkdir -p /srv/restore-test
sudo rsync -aHAX /mnt/backup/app/ /srv/restore-test/
sudo diff -qr /srv/app /srv/restore-test

Check ownership, permissions, links, application configuration, and one real read workflow. For a database, restore into a separate database name and run integrity checks. Record elapsed time so the recovery-time target is based on evidence.

Try this on one small backup: restore it into isolation, compare file counts and checksums, start the service if safe, and record whether the result met RPO and RTO.