Losing the application server; losing the database or its provider; a failure at an email provider; a mistake in a release; a key being lost.
Backups: an encrypted copy of the whole database is taken every night and kept for 30 days on the application server in London, separately from the database provider in Ireland, which keeps its own copies. Each night’s backup is restored into a scratch database straight after it is taken, to prove it opens, and a failure is emailed to us.
The application: the code is held in version control off the server, so the application can be rebuilt on a new server from it and a backup.
Email: if Amazon SES fails, email is sent through a second provider until it recovers, so an alert still reaches you.
Releases: a backup is taken before every production deployment, and a release can be rolled back.
Keys: the encryption key is held in two places, the server and an encrypted offline copy held by the company’s technical lead. Without it, backups cannot be opened, which is the point.
Status: the health of the platform is published at monitoring.walterssentinel.com/status.
Point to recover to: the last nightly backup, or the database provider’s own copies where they are more recent.
Our aim for a total loss of the application server is to have the service running again on a new server within one working day. This is an aim, not a contractual commitment.
Restoring a backup is tested every night, automatically. Building the service on a new, empty server follows the same steps as setting up a dedicated deployment, which has its own verification suite that starts from an empty database.