Recovery drill
Test the service you must recover, not only the file you can export
Recovery objectives connect business tolerance to technical scope, order, integrity and communication.
Recovery objective
Define maximum tolerable data loss and service interruption.
- Decide
- RPO, RTO and critical journeys
- Risk
- Vendor default does not match business need
- Evidence
- Approved objectives and drill target
Coverage
Inventory state needed for a functioning product.
- Decide
- Database, objects, identity config, DNS and secrets metadata
- Risk
- Database restores but uploads or keys are missing
- Evidence
- Dependency-aware recovery inventory
Isolation and access
Keep backups protected and restoration controlled.
- Decide
- Encryption, access, region and retention
- Risk
- Backup becomes an unmonitored copy of production
- Evidence
- Least-privilege access and audit trail
Restore procedure
Recreate service in a clean, documented order.
- Decide
- Environment, sequence, credentials and owners
- Risk
- Restore depends on one person’s memory
- Evidence
- Timed runbook executed by another operator
Integrity and return
Prove recovered state is complete, current enough and safe to reopen.
- Decide
- Counts, checksums, journeys and reconciliation
- Risk
- Service opens with silently inconsistent state
- Evidence
- Acceptance checks and gap log
Operating principle
Recovery is a cross-system product capability
The database rarely contains the whole operating product. Authentication settings, object storage, DNS, scheduled jobs, provider configuration and encryption access can determine whether restored records are usable.
Build the runbook around one critical user journey. Recover its dependencies in a clean environment, verify integrity and record actual time and data loss against the approved objectives.
- Test in isolation
- Protect backup access
- Measure actual recovery time
- Record unrecovered dependencies
Applied example
Recovery example: data returns, product does not
A provider successfully restores the database to a new project. Users still cannot sign in because identity configuration and encryption material were not included in the recovery inventory.
- The database backup itself worked
- The service dependency map was incomplete
- Recovery acceptance was defined too narrowly
- Configuration ownership became the critical path
Plain answers
Questions to resolve before shipping
How often should we test restore?+
Choose a cadence from change rate, consequence and recovery objectives, and repeat after material architecture or provider changes.
Is provider point-in-time recovery enough?+
It may be a strong component, but verify coverage, retention, access, regional assumptions and the complete service restoration path.
Can production data be used in a drill?+
Use an approved isolated approach that minimizes sensitive data and follows privacy, security and retention obligations.