Docs / Recovery runbooks

Recovery runbooks

Follow these steps in order on the day you need them, even if you have never run safegrd before. An automated test runs every command here, in this order, on each change to SafeGrd. The first runbook runs on a new host enrolled into the same project. The others run on a machine that has only the key file and read access to the bucket.

Which one you follow depends on the host's key, which the console shows on each host. With a SafeGrd-managed key, start at the host is gone. With a customer-managed key, start at before you start. If you cannot reach SafeGrd, start at without SafeGrd.

Which runbook to follow. If you can reach SafeGrd and the host used a SafeGrd-managed key, follow the host is gone: enroll a new host and restore. If the host used a customer-managed key, start at before you start. If you cannot reach SafeGrd, read restoring without SafeGrd, which leads to before you start. From before you start, follow runbooks 1 to 5 by what is gone. Can you reach SafeGrd? yes no Which key? shown on each host Without SafeGrd key file, your bucket SafeGrd- managed customer-managed The host is gone enroll a new host Before you start key file, bucket, CLI Runbooks 1 to 5 by what is gone
To restore without SafeGrd, you need the key file plus the backups in your own bucket or an export of hosted backups. That holds whichever key the host used.

These steps need a CLI newer than v0.0.2. Earlier versions cannot find a snapshot without its node id. Check with safegrd --version, and install the latest if yours is older.

With a SafeGrd-managed key: the host is gone

A SafeGrd-managed key is the default. SafeGrd keeps each host's key sealed and releases it only to your organization's enrolled hosts, so a new host can read what the old one wrote, even after you remove the old host from the console. You need a machine that can reach the database or directory you are restoring into, and a login to the console. You do not need a key file or bucket credentials.

  1. Install and enroll the new machine:
    curl -fsSL https://safegrd.dev/install.sh | sh
    Answer Y to connect it, then approve the sign-in in a browser on any device. If the organization has more than one project, enrollment asks which one. Choose the old host's project, so the new host uses the same storage. Without a terminal, the machine joins the default project. In that case, run safegrd enroll --project SLUG instead (the slug is on the console's Projects page).
  2. Find the snapshot to restore with safegrd list, or on the console's Snapshots page, where each row names the surface and when it was taken. Copy its ID.
  3. Create a new, empty database and restore into it:
    safegrd restore --snapshot SNAPSHOT_ID --target "postgres://user:pass@host:5432/recovered"
    For a directory or a mailbox, use --target-dir ./restored, as in Runbook 2 and Runbook 3. The restore prints Decryption key: SG:…, SafeGrd-managed key of org … (fetched, not stored). That means SafeGrd released this snapshot's key to this host for this restore, and the host did not write it to disk. Before it reports success, the restore checks every table against the digest and row count recorded at backup time.
  4. Check what was released, and to which host: in the console, Projects, the project's Storage, under Key access record. The row names this host, the key it was given and the snapshot.
  5. Back up the restored database from this host: in the console, click Add surface, then run the command it shows on this machine.
  6. Retire the old host: under Hosts, open its Details, then Edit, and choose Retire. Its snapshots stay in storage, and SafeGrd keeps its key, so they still restore. Until you retire it, the console marks it Silent after three missed check-ins (at least 15 minutes) and alerts the organization's owners.

The same steps work when the project's backups go to your own bucket (set under the project's Storage in Projects). SafeGrd gives the enrolled host the bucket's credentials the same way.

Restoring without SafeGrd

A restore needs the snapshot's bytes and its key. The CLI can read both with no SafeGrd account or server, so what you do depends on where they are.

Backups inWhat to do
Your own bucketFollow before you start and set up the recovery machine, then the runbook for what you lost.
Hosted storageKeep a copy in your own bucket with safegrd export (below), and restore from that copy the same way. Hosted storage is read and written through SafeGrd: each upload gets its URLs from SafeGrd's server, so backups to hosted storage run while the server answers. A backup to your own bucket needs only the bucket.

With either kind of key, enrollment writes the key file to ~/.safegrd/keys/daemon.key on the host that took the backups. Keep a copy of it with your copy of the backups.

Keep a copy of hosted backups

Run safegrd export on the host on a schedule. It copies each snapshot still encrypted, checks it against the digest recorded at backup time and keeps its lock date. It skips snapshots already in the bucket, so each run copies only what is new. The bucket needs Object Lock (or pass --to-worm-mode NONE). The credentials come from the standard AWS environment variables. A daily crontab line:

0 3 * * * AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... safegrd export --to-bucket your-bucket --to-endpoint https://your-s3-endpoint

Or let the daemon run it, with no crontab line. Name the bucket in daemon.export_to in the host's config, and the daemon copies to it daily after its backups. The console shows on the host when the last copy was made. The credentials take env:VAR or file:/path.

daemon:
  export_to:
    bucket: "your-bucket"
    endpoint: "https://your-s3-endpoint"
    access_key_id: "env:EXPORT_ACCESS_KEY_ID"
    secret_access_key: "env:EXPORT_SECRET_ACCESS_KEY"
    schedule: "@daily"

Leave out --to-endpoint (or endpoint) on AWS. The copy is filed under safegrd/snapshots, the prefix the recovery config below uses. --to-dir copies to a directory instead, and Runbook 4 restores from that.

With a customer-managed key: before you start

You need three things:

  1. The private key file, which starts AGE-SECRET-KEY-1. It is usually ~/.safegrd/keys/daemon.key on the host that took the backups. Keep a copy offline. With a customer-managed key, it is the one file that decrypts these backups.
  2. Read access to the bucket: its name, its region, the endpoint if it is not AWS, and an access key that can list and read it.
  3. The safegrd CLI on the machine you are restoring to: curl -fsSL https://safegrd.dev/install.sh | SAFEGRD_NO_SETUP=1 sh. SAFEGRD_NO_SETUP=1 installs the binary and stops there. Enrolling this machine would generate a new key, and a restore needs the old one.

You do not need SafeGrd, the old host or its config. If the host that took the backups still works, skip the recovery config below and run the same commands there.

Set up the recovery machine

  1. Copy the key file to this machine and make it readable only by you:
    chmod 600 ./safegrd.key
  2. Find the bucket's prefix. With a wrong prefix, the CLI finds no snapshots at all. If the bucket was set up in the console, the old host's config.yaml does not have it, so read it from the bucket. An incremental backup is stored under PREFIX/NODE_ID/repo/, and a full copy as PREFIX/NODE_ID/SNAPSHOT_ID.safegrd. The prefix is everything before the node id. It is safegrd/snapshots unless someone changed it.
    aws s3 ls s3://your-bucket/ --recursive | grep -E '/repo/|\.safegrd$' | head -3
    Add --endpoint-url https://your-s3-endpoint when it is not AWS. Write the prefix down now, while you can also check it under the project's Storage in Projects.
  3. Write recovery.yaml. Leave out endpoint on AWS. Set force_path_style: true only for MinIO.
    storage:
      type: "s3"
      bucket: "your-bucket"
      region: "us-east-1"
      endpoint: "https://your-s3-endpoint"
      prefix: "safegrd/snapshots"
    encryption:
      key_path: "./safegrd.key"
    safegrd reads a config only if no one but its owner can read it:
    chmod 600 recovery.yaml
  4. Pass the bucket credentials through the environment, so no secret sits in the file:
    export AWS_ACCESS_KEY_ID=...
    export AWS_SECRET_ACCESS_KEY=...
  5. List what is there. Each row names the snapshot, the host it was filed under, what it holds and when it can be deleted. Incremental file backups appear in a second table, by month. You do not need to know the node id, because restore finds it.
    safegrd --config recovery.yaml list
  6. Before you touch anything, check that the snapshot you picked decrypts and is whole. This writes nothing anywhere:
    safegrd --config recovery.yaml verify --snapshot SNAPSHOT_ID --in-memory
    If it fails, stop. Try the next-newest snapshot, and see Troubleshooting.

Runbook 1: the database is gone

  1. Create a new, empty database to restore into. Do not restore over a database that still holds data you may need. Restore into a new one, check it, then switch your application over.
    createdb recovered
  2. Restore into it:
    safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target "postgres://user:pass@host:5432/recovered"
    Before it reports success, the restore checks the snapshot's digest. If it could check it only against the sidecar, it says so, which is expected when SafeGrd is unreachable. It creates any role the schema names that the new cluster lacks, with its password, and prints each one. Run it as a superuser so every owner and grant comes back (roles, owners and grants).
  3. Check the tables and row counts it printed against what you expect, then run a query your application depends on.
  4. Point your application at recovered. Once it is serving, take a fresh backup from it: safegrd backup --database-url ...
prod-app-01
$ safegrd restore --snapshot snap-20261018-024047-450e5d --target env:SCRATCH_DB
Restoring snap-20261018-024047-450e5d (epoch e202610-06bca287, shop) into postgres://postgres:xxxxx@db.internal:5432/shop_restored?sslmode=verify-full&sslrootcert=/etc/ssl/rec-ca.crt
   Content root:   matches the remote server's record from backup time
Restored 3 tables, 70,240 rows, from snap-20261018-024047-450e5d in 147ms. Every table matched its SHA-256 and its row count.
What step 2 prints on an enrolled host. The content root is checked against SafeGrd's record from backup time, and every table against its SHA-256 and row count.

Runbook 2: a directory tree is gone

  1. Restore into a new directory, not over the old path:
    safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target-dir ./restored
  2. Read the last lines it prints, which say what it restored and what it cannot restore. Ownership comes back only when you run the restore as root. Hard links, extended attributes and ACLs are not restored (the full list).
  3. If you need ownership back, run the same command with sudo, pointing at the same config and key.
  4. Check a few files you know, then move the directory into place.

A restore never writes outside --target-dir. If the target already has a symlink where the backup has a file or directory, the restore stops and names the symlink instead of following it.

Runbook 3: a mailbox is gone

  1. Restore the messages to disk:
    safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target-dir ./mail
  2. Each mailbox folder becomes a directory and each message a standard .eml file, readable only by you. Any mail client can open them. A Gmail mailbox comes back as one All Mail directory, with each message's labels in .safegrd-email-manifest.json (Gmail labels).
  3. To put them back on a server, import the directory with your mail system's own import tool, or drag the .eml files into the mailbox from a desktop client connected over IMAP.

Runbook 4: restore from an export

You can restore from a safegrd export --to-dir copy with only the key, and no bucket. The export writes a single-archive snapshot as NODE/SNAPSHOT_ID.safegrd, with its metadata beside it in SNAPSHOT_ID.meta.json. It writes an incremental backup (the default) as its month of packs under NODE/repo/. The snapshot ID is the one safegrd backup printed, or the name of a file under snapshots/ in the export.

  1. Restore from the export directory, or from one file in it:
    safegrd restore --from ./export --snapshot SNAPSHOT_ID --target-dir ./restored
    safegrd restore --from ./export/NODE/SNAPSHOT_ID.safegrd --target-dir ./restored
    The key comes from --key-path, --private-key or the config, as in the runbooks above. The restore checks the digest against SafeGrd's record when the host can reach SafeGrd, and against the .meta.json beside the file when it cannot.
  2. For a database snapshot, use --target instead of --target-dir, as in Runbook 1.

Without SafeGrd

A .safegrd file is zstd-compressed data encrypted to your Age key, so age and zstd open it:

age -d -i safegrd.key SNAPSHOT_ID.safegrd | zstd -d > payload

For a files or email snapshot, payload is a tar archive. tar -xf payload gives back the directory tree, or the .eml files and their manifest.

A PostgreSQL snapshot is a tar archive too: the schema as SQL from pg_dump, then each table's rows in PostgreSQL's binary COPY format. pg_restore does not read it, but psql can load it once it is written out as SQL files. safegrd restore --snapshot ID --to-sql DIR writes those files, with a load.sql that runs them in order. psql -f load.sql then loads them into an empty database with no SafeGrd involved.

This route checks nothing, so compare sha256sum payload with sha256_checksum in the .meta.json yourself. It works only for single-archive snapshots: mailboxes, and databases or files backed up with format: tar. An incremental backup is a set of packs, indexes and directory listings, not one archive. For those, safegrd export --to-dir copies a whole month, and safegrd restore --from reads the copy with the key and no bucket.

Runbook 5: one file, as it was on a given day

Use this for a directory tree backed up incrementally, the default for files. To restore the whole tree, follow Runbook 2.

  1. List the kept versions of the file. Paths are relative to /, with no leading slash:
    safegrd --config recovery.yaml find etc/nginx/nginx.conf
    Each version is one distinct content of the file, numbered from the oldest, with the first and last snapshot that holds it. For a directory, find lists everything below it. find --deleted var/www/uploads/ lists only files the newest backup no longer holds.
  2. Restore the version whose dates bracket the day you want, into a new directory:
    safegrd --config recovery.yaml restore --path etc/nginx/nginx.conf --version 2 --target-dir ./restored
    Or restore everything under a directory as it was in one snapshot, with the id from find or list:
    safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --path 'var/www/uploads/**' --target-dir ./restored
    The file comes back under the target at the path a whole-tree restore would give it (from the root for a surface of one directory, from / otherwise). It keeps the permissions and times it had in that snapshot. The restore downloads only that file's chunks and checks every byte against the digest recorded at backup time.
  3. Compare sha256sum of the restored file with the SHA-256 find printed, then copy it into place.

Rehearse it before you need it

Rehearse a restore every quarter, and before a compliance audit, following this page:

  1. Use a machine that is not your production host, such as a laptop or a fresh VM.
  2. Put only the key file and the read-only bucket credentials on it, and follow Set up the recovery machine.
  3. Run Runbook 1 into a scratch database, Runbook 2 into a scratch directory, and Runbook 5 for one file from a month ago.
  4. Write down the date, the snapshot id and how long it took. That is your real recovery time. A Fire Drill proves each snapshot restores, and this rehearsal proves that you can restore it. The console’s restore time comes from drills, some of them run on SafeGrd’s machines. This one is timed on your own hardware.
  5. Delete the scratch copies, and remove the key file from the machine if it is not kept secure.