Recovery runbooks
Follow these steps in order on the day you need them, even if you have never run
safegrd before. An automated test runs every command here, in this order, on each
change to SafeGrd. The first runbook runs on a new host enrolled into the
same project. The others run on a machine that has only the key file and read access to the
bucket.
Which one you follow depends on the host's key, which the console shows on each host. With a SafeGrd-managed key, start at the host is gone. With a customer-managed key, start at before you start. If you cannot reach SafeGrd, start at without SafeGrd.
These steps need a CLI newer than v0.0.2. Earlier versions cannot find a snapshot without
its node id. Check with safegrd --version, and install the latest if yours is
older.
With a SafeGrd-managed key: the host is gone
A SafeGrd-managed key is the default. SafeGrd keeps each host's key sealed and releases it only to your organization's enrolled hosts, so a new host can read what the old one wrote, even after you remove the old host from the console. You need a machine that can reach the database or directory you are restoring into, and a login to the console. You do not need a key file or bucket credentials.
- Install and enroll the new machine:
curl -fsSL https://safegrd.dev/install.sh | shAnswer
Yto connect it, then approve the sign-in in a browser on any device. If the organization has more than one project, enrollment asks which one. Choose the old host's project, so the new host uses the same storage. Without a terminal, the machine joins the default project. In that case, runsafegrd enroll --project SLUGinstead (the slug is on the console's Projects page). - Find the snapshot to restore with
safegrd list, or on the console's Snapshots page, where each row names the surface and when it was taken. Copy its ID. - Create a new, empty database and restore into it:
safegrd restore --snapshot SNAPSHOT_ID --target "postgres://user:pass@host:5432/recovered"For a directory or a mailbox, use
--target-dir ./restored, as in Runbook 2 and Runbook 3. The restore prints Decryption key: SG:…, SafeGrd-managed key of org … (fetched, not stored). That means SafeGrd released this snapshot's key to this host for this restore, and the host did not write it to disk. Before it reports success, the restore checks every table against the digest and row count recorded at backup time. - Check what was released, and to which host: in the console, Projects, the project's Storage, under Key access record. The row names this host, the key it was given and the snapshot.
- Back up the restored database from this host: in the console, click Add surface, then run the command it shows on this machine.
- Retire the old host: under Hosts, open its Details, then Edit, and choose Retire. Its snapshots stay in storage, and SafeGrd keeps its key, so they still restore. Until you retire it, the console marks it Silent after three missed check-ins (at least 15 minutes) and alerts the organization's owners.
The same steps work when the project's backups go to your own bucket (set under the project's Storage in Projects). SafeGrd gives the enrolled host the bucket's credentials the same way.
Restoring without SafeGrd
A restore needs the snapshot's bytes and its key. The CLI can read both with no SafeGrd account or server, so what you do depends on where they are.
| Backups in | What to do |
|---|---|
| Your own bucket | Follow before you start and set up the recovery machine, then the runbook for what you lost. |
| Hosted storage | Keep a copy in your own bucket with
safegrd export (below), and restore from that copy the same way. Hosted storage
is read and written through SafeGrd: each upload gets its URLs from SafeGrd's server, so
backups to hosted storage run while the server answers. A backup to your own bucket needs
only the bucket. |
With either kind of key, enrollment writes the key file to
~/.safegrd/keys/daemon.key on the host that took the backups. Keep a copy of it
with your copy of the backups.
Keep a copy of hosted backups
Run safegrd export on the host on a schedule. It copies each snapshot still
encrypted, checks it against the digest recorded at backup time and keeps its lock date. It
skips snapshots already in the bucket, so each run copies only what is new. The bucket needs
Object Lock (or pass --to-worm-mode NONE). The credentials come from the standard
AWS environment variables. A daily crontab line:
Or let the daemon run it, with no crontab line. Name the bucket in
daemon.export_to in the host's config, and the daemon copies to it daily after its
backups. The console shows on the host when the last copy was made. The credentials take
env:VAR or file:/path.
Leave out --to-endpoint (or endpoint) on AWS. The copy is filed under
safegrd/snapshots, the prefix the recovery config below uses.
--to-dir copies to a directory instead, and
Runbook 4 restores from that.
With a customer-managed key: before you start
You need three things:
- The private key file, which starts
AGE-SECRET-KEY-1. It is usually~/.safegrd/keys/daemon.keyon the host that took the backups. Keep a copy offline. With a customer-managed key, it is the one file that decrypts these backups. - Read access to the bucket: its name, its region, the endpoint if it is not AWS, and an access key that can list and read it.
- The
safegrdCLI on the machine you are restoring to:curl -fsSL https://safegrd.dev/install.sh | SAFEGRD_NO_SETUP=1 sh.SAFEGRD_NO_SETUP=1installs the binary and stops there. Enrolling this machine would generate a new key, and a restore needs the old one.
You do not need SafeGrd, the old host or its config. If the host that took the backups still works, skip the recovery config below and run the same commands there.
Set up the recovery machine
- Copy the key file to this machine and make it readable only by you:
chmod 600 ./safegrd.key
- Find the bucket's
prefix. With a wrong prefix, the CLI finds no snapshots at all. If the bucket was set up in the console, the old host'sconfig.yamldoes not have it, so read it from the bucket. An incremental backup is stored underPREFIX/NODE_ID/repo/, and a full copy asPREFIX/NODE_ID/SNAPSHOT_ID.safegrd. The prefix is everything before the node id. It issafegrd/snapshotsunless someone changed it.aws s3 ls s3://your-bucket/ --recursive | grep -E '/repo/|\.safegrd$' | head -3Add--endpoint-url https://your-s3-endpointwhen it is not AWS. Write the prefix down now, while you can also check it under the project's Storage in Projects. - Write
recovery.yaml. Leave outendpointon AWS. Setforce_path_style: trueonly for MinIO.safegrd reads a config only if no one but its owner can read it:storage:type: "s3"bucket: "your-bucket"region: "us-east-1"endpoint: "https://your-s3-endpoint"prefix: "safegrd/snapshots"encryption:key_path: "./safegrd.key"chmod 600 recovery.yaml - Pass the bucket credentials through the environment, so no secret sits in the file:
export AWS_ACCESS_KEY_ID=...export AWS_SECRET_ACCESS_KEY=...
- List what is there. Each row names the snapshot, the host it was filed under, what it
holds and when it can be deleted. Incremental file backups appear in a second table, by
month. You do not need to know the node id, because restore finds it.
safegrd --config recovery.yaml list
- Before you touch anything, check that the snapshot you picked decrypts and is whole.
This writes nothing anywhere:
safegrd --config recovery.yaml verify --snapshot SNAPSHOT_ID --in-memoryIf it fails, stop. Try the next-newest snapshot, and see Troubleshooting.
Runbook 1: the database is gone
- Create a new, empty database to restore into. Do not restore over a
database that still holds data you may need. Restore into a new one, check it, then switch
your application over.
createdb recovered
- Restore into it:
safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target "postgres://user:pass@host:5432/recovered"Before it reports success, the restore checks the snapshot's digest. If it could check it only against the sidecar, it says so, which is expected when SafeGrd is unreachable. It creates any role the schema names that the new cluster lacks, with its password, and prints each one. Run it as a superuser so every owner and grant comes back (roles, owners and grants).
- Check the tables and row counts it printed against what you expect, then run a query your application depends on.
- Point your application at
recovered. Once it is serving, take a fresh backup from it:safegrd backup --database-url ...
$ safegrd restore --snapshot snap-20261018-024047-450e5d --target env:SCRATCH_DB Restoring snap-20261018-024047-450e5d (epoch e202610-06bca287, shop) into postgres://postgres:xxxxx@db.internal:5432/shop_restored?sslmode=verify-full&sslrootcert=/etc/ssl/rec-ca.crt Content root: matches the remote server's record from backup time Restored 3 tables, 70,240 rows, from snap-20261018-024047-450e5d in 147ms. Every table matched its SHA-256 and its row count.
Runbook 2: a directory tree is gone
- Restore into a new directory, not over the old path:
safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target-dir ./restored
- Read the last lines it prints, which say what it restored and what it cannot restore. Ownership comes back only when you run the restore as root. Hard links, extended attributes and ACLs are not restored (the full list).
- If you need ownership back, run the same command with
sudo, pointing at the same config and key. - Check a few files you know, then move the directory into place.
A restore never writes outside --target-dir. If the target already has a
symlink where the backup has a file or directory, the restore stops and names the symlink
instead of following it.
Runbook 3: a mailbox is gone
- Restore the messages to disk:
safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target-dir ./mail
- Each mailbox folder becomes a directory and each message a standard
.emlfile, readable only by you. Any mail client can open them. A Gmail mailbox comes back as oneAll Maildirectory, with each message's labels in.safegrd-email-manifest.json(Gmail labels). - To put them back on a server, import the directory with your mail system's own import
tool, or drag the
.emlfiles into the mailbox from a desktop client connected over IMAP.
Runbook 4: restore from an export
You can restore from a safegrd export --to-dir copy with only the key, and no
bucket. The export writes a single-archive snapshot as NODE/SNAPSHOT_ID.safegrd,
with its metadata beside it in SNAPSHOT_ID.meta.json. It writes an incremental
backup (the default) as its month of packs under NODE/repo/. The snapshot ID is the
one safegrd backup printed, or the name of a file under snapshots/ in
the export.
- Restore from the export directory, or from one file in it:
The key comes fromsafegrd restore --from ./export --snapshot SNAPSHOT_ID --target-dir ./restoredsafegrd restore --from ./export/NODE/SNAPSHOT_ID.safegrd --target-dir ./restored
--key-path,--private-keyor the config, as in the runbooks above. The restore checks the digest against SafeGrd's record when the host can reach SafeGrd, and against the.meta.jsonbeside the file when it cannot. - For a database snapshot, use
--targetinstead of--target-dir, as in Runbook 1.
Without SafeGrd
A .safegrd file is zstd-compressed data encrypted to your Age key, so
age and zstd open it:
For a files or email snapshot, payload is a tar archive. tar -xf payload
gives back the directory tree, or the .eml files and their manifest.
A PostgreSQL snapshot is a tar archive too: the schema as SQL from pg_dump, then
each table's rows in PostgreSQL's binary COPY format. pg_restore does
not read it, but psql can load it once it is written out as SQL files.
safegrd restore --snapshot ID --to-sql DIR writes those files, with a
load.sql that runs them in order. psql -f load.sql then loads them
into an empty database with no SafeGrd involved.
This route checks nothing, so compare sha256sum payload with
sha256_checksum in the .meta.json yourself. It works only for
single-archive snapshots: mailboxes, and databases or files backed up with
format: tar. An incremental backup is a set of packs, indexes and directory
listings, not one archive. For those, safegrd export --to-dir copies a whole month,
and safegrd restore --from reads the copy with the key and no bucket.
Runbook 5: one file, as it was on a given day
Use this for a directory tree backed up incrementally, the default for files. To restore the whole tree, follow Runbook 2.
- List the kept versions of the file. Paths are relative to
/, with no leading slash:safegrd --config recovery.yaml find etc/nginx/nginx.confEach version is one distinct content of the file, numbered from the oldest, with the first and last snapshot that holds it. For a directory,findlists everything below it.find --deleted var/www/uploads/lists only files the newest backup no longer holds. - Restore the version whose dates bracket the day you want, into a new directory:
safegrd --config recovery.yaml restore --path etc/nginx/nginx.conf --version 2 --target-dir ./restoredOr restore everything under a directory as it was in one snapshot, with the id from
findorlist:safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --path 'var/www/uploads/**' --target-dir ./restoredThe file comes back under the target at the path a whole-tree restore would give it (from the root for a surface of one directory, from/otherwise). It keeps the permissions and times it had in that snapshot. The restore downloads only that file's chunks and checks every byte against the digest recorded at backup time. - Compare
sha256sumof the restored file with the SHA-256findprinted, then copy it into place.
Rehearse it before you need it
Rehearse a restore every quarter, and before a compliance audit, following this page:
- Use a machine that is not your production host, such as a laptop or a fresh VM.
- Put only the key file and the read-only bucket credentials on it, and follow Set up the recovery machine.
- Run Runbook 1 into a scratch database, Runbook 2 into a scratch directory, and Runbook 5 for one file from a month ago.
- Write down the date, the snapshot id and how long it took. That is your real recovery time. A Fire Drill proves each snapshot restores, and this rehearsal proves that you can restore it. The console’s restore time comes from drills, some of them run on SafeGrd’s machines. This one is timed on your own hardware.
- Delete the scratch copies, and remove the key file from the machine if it is not kept secure.