Run a disaster recovery exercise

Everything in this course converges here. A restore drill tests one dataset; a disaster recovery exercise tests the whole story. The final exercise begins as if the original machine is gone. Not “unavailable for a bit” — gone, along with every convenient shortcut it held: shell history, SSH keys, that one script only it had.

The discipline that makes it real: use a clean disposable system and the same recovery boundaries a real incident would require. A fresh VM or spare machine, your written runbook, your protected credentials. Nothing from memory. Every time you catch yourself typing something the runbook doesn’t say, you’ve found a gap — the exercise is working.

Follow the runbook in order

1. establish clean recovery host
2. install verified tools
3. obtain protected credentials
4. select known-good recovery point
5. restore data and configuration
6. verify application behavior
7. record time and gaps

Each step fails in its own way, which is why each one is on the list. Step 2 catches undocumented tool versions. Step 3 is where circular credential dependencies surface. Step 4 forces you to choose a snapshot and justify it, not reflexively grab latest. Step 6 is the honest finish line: files on disk mean nothing until the application starts and behaves correctly against them.

Keep a timer running and a notes file open. Timestamps per step, every surprise, every improvisation:

14:02  host ready
14:19  restic installed, version pinned in runbook? NO - gap
14:31  credentials located (map was accurate)
15:58  restore complete, app starts, orders count matches

Score it against your targets

Compare the measured recovery point and time with RPO and RTO. If the exercise took four hours against a one-hour RTO, you now know — cheaply, calmly — that the target is fiction. Either improve the process or renegotiate the target. Then turn every manual guess into an updated instruction or automation, and schedule the next exercise, because runbooks rot as systems change.

One safety boundary throughout: keep the exercise isolated from production names, credentials, and writable storage until the restored system is verified. A recovered application with stale config can email real customers or write to the real database. The exercise should prove recovery works — quietly.