# SafeGrd documentation Every page of https://safegrd.dev/docs as Markdown. Each page alone is at its address with .md added. --- Source: https://safegrd.dev/docs # Documentation You install SafeGrd on a host that can reach your database. It encrypts each backup on that host, writes it to a locked bucket, and restores it on a schedule to test it. Your data never passes through SafeGrd's server. If you set a surface to drill on SafeGrd, it is restored on a machine started for that drill and destroyed when it ends. ## Quickstart You can try it without an account. Once it is installed, `safegrd try postgres://…` backs up one database and restores it as a check. Run `safegrd signup` to keep the backup ([Try it before you sign up](https://safegrd.dev/docs/install#try)). Three commands get you a tested backup. The first prints a sign-in link you can open on any device, so it works over SSH too. ``` # 1. install, log in, and enroll this host (generates your keypair) curl -fsSL https://safegrd.dev/install.sh | sh # listed under its hostname; add SAFEGRD_NODE_NAME=web-01 before sh to choose the name # 2. take an encrypted backup safegrd backup --database-url "$DATABASE_URL" --retention-days 14 # 3. check that it restores safegrd verify --snapshot snap-1b094c0af8 ``` ### With a customer-managed key, keep a copy of ~/.safegrd/keys/daemon.key. It is the key that opens your backups. With a [SafeGrd-managed key](https://safegrd.dev/docs/install#custody), SafeGrd keeps it sealed for you. If you have no account, or the host cannot reach SafeGrd, run `safegrd init` instead. It sets up a host that backs up, restores and verifies on its own. ## What the console shows The other tabs are covered under [Fire Drills](https://safegrd.dev/docs/verify) and [Account](https://safegrd.dev/docs/account). ## Where to go next 01 Install and enroll Install the CLI, enroll a host and choose how its key is managed. 02 Command reference Every command, flag and environment variable. 03 Surfaces What each database, file tree and mailbox backup captures. 04 Storage and retention Use your own bucket or ours, with Object Lock and retention. 05 Threat Shield Get an alert when tables or rows vanish. The last good snapshot is kept. 06 Daemon Schedules, the service and how it retries. 07 Restore and Fire Drills Restore a backup, or test a restore away from production. 08 The attestation record The signed record of every drill, checkable offline. 09 Security and key custody What SafeGrd holds and what it never receives. 10 Organizations and billing Projects, members, tokens and billing. 11 AI agents (MCP) Let a coding agent check, take and test backups. It cannot delete one. 12 Troubleshooting Common errors and their fixes. --- Source: https://safegrd.dev/docs/install # Install and enroll Install SafeGrd on a host that can already reach what you want to back up. It compresses and encrypts each backup there, so only encrypted data leaves the machine. Setup has two parts. The first runs on the host and generates your keypair. The second is optional: it registers the host with SafeGrd so the console can track it. ## Install the binary ``` # macOS and Linux, amd64 and arm64 curl -fsSL https://safegrd.dev/install.sh | sh # the host is listed under its hostname; to choose the name: curl -fsSL https://safegrd.dev/install.sh | SAFEGRD_NODE_NAME=web-01 sh ``` The installer detects your platform, downloads the matching release and checks it against the release's `checksums.txt`. It puts `safegrd` in `/usr/local/bin`, or in `~/.local/bin` when it cannot use sudo. To read the script before you run it, fetch [the script itself](https://safegrd.dev/install.sh). It is the same file as [install.sh in the CLI repository](https://github.com/safegrd/cli/blob/main/install.sh), and a test fails the build if the two ever differ. The console's setup wizard (**Add surface**) prints the same command with the project you picked already filled in. You do not need a flag to put the host in that project. ### What it asks in a terminal In a terminal, the installer offers to connect the host straight away. One command takes you from nothing to an enrolled host: - You log in from a browser. The installer prints a code and a link. Open the link in a browser on _any_ device, check that it shows the same code, and approve. The page shows the address the login came from, so you can tell it is yours. It asks for your password only if you signed in to the console more than 30 minutes ago. On a server you reached over SSH or PuTTY, use the browser on your laptop. The CLI does not try to open one on the server. - The installer enrolls the host: it generates the keypair and registers the host. SafeGrd keeps the private key sealed unless you set `SAFEGRD_KEY_CUSTODY=local` or chose that for this host in the console's setup ([below](https://safegrd.dev/docs/install#custody)). The install ends with the next steps that `enroll` prints for this host: - `safegrd doctor` checks that every surface opens. - `safegrd daemon run --once` takes the first backup. - `sudo safegrd daemon install` keeps the daemon running. Nothing backs up on a schedule until the daemon runs. If the host has nothing to protect yet, `enroll` first tells you how to name it: click **Add surface** on the host's row under Hosts, then run `safegrd claim`. The console's setup shows the same steps. Without a terminal (CI, cloud-init, a cron job), the installer installs the binary and stops. If you run it again on a host that is already enrolled, it upgrades the binary and changes nothing else. To add surfaces to that host, name them in the console with **Add surface** on its row under Hosts, then run `safegrd claim` on the host ([details](https://safegrd.dev/docs/surfaces#adding)). The installer keeps each release in `/usr/local/lib/safegrd/versions` (`~/.local/lib/safegrd` without sudo) and points `/usr/local/bin/safegrd` at the one in use. This is how you can update the host from the console and go back to the release it ran before ([updates from the console](https://safegrd.dev/docs/daemon#updates)). If you set `SAFEGRD_INSTALL_DIR`, the binary there is a plain copy. The console cannot update it unless you also set `SAFEGRD_LIB_DIR`. ### Options Options are environment variables. Put them before the `sh` after the pipe: `curl -fsSL https://safegrd.dev/install.sh | VERSION=v0.0.3 sh`. If you put one before `curl`, it is set for `curl` and the script never sees it. | Variable | Effect | | --- | --- | | VERSION | Release tag to install. Default: the latest release. | | SAFEGRD_INSTALL_DIR | Where the binary goes. | | SAFEGRD_LIB_DIR | With `SAFEGRD_INSTALL_DIR`: where each release is kept. The binary in `SAFEGRD_INSTALL_DIR` links to it, and the console can update the host. | | SAFEGRD_NO_SETUP=1 | Install only, without the offer to log in and enroll. Use it on a machine you are restoring to. | | SAFEGRD_FORCE_INSTALL=1 | Download and install even if that release is already installed. Without it, a host that already has the release goes straight to setup. | | SAFEGRD_PROJECT | Project ID or slug to enroll this host into. | | SAFEGRD_NODE_NAME | The name the console shows for this host. Default: the machine's hostname, without its domain. | | SAFEGRD_KEY_CUSTODY | `local` keeps this host's key on the host (a customer-managed key). Without it, SafeGrd keeps the key sealed. | | SAFEGRD_CLAIM | The claim code the console shows for this host. The host enrolls into that project, with the storage and surfaces you chose there. | | SAFEGRD_TOKEN | A token from Tokens in the console (`sg_pat_…`). The host enrolls without a terminal or a browser login. Use it for CI, cloud-init or `ssh host 'curl … \| sh'`. | There is no Windows build. The daemon runs on Linux and macOS. On a Windows host, run it in the Docker image. ### Other ways to install ``` # Homebrew, macOS and Linux: the same release archives brew install safegrd/tap/safegrd # Go go install github.com/safegrd/cli/cmd/safegrd@latest # Docker: the daemon as a sidecar, with its config and key in a volume docker run --rm -it -v safegrd:/home/safegrd/.safegrd ghcr.io/safegrd/cli enroll ``` The Docker image includes `pg_dump` 18, the MariaDB client, the MongoDB tools and a PostgreSQL 18 server. With that server, a drill can restore a database into a throwaway cluster on the volume. The container runs as its own user. To run it beside a database, use [compose.yaml](https://github.com/safegrd/cli/blob/main/packaging/compose/compose.yaml). To run it on Kubernetes, use the [Helm chart](https://github.com/safegrd/cli/tree/main/packaging/helm/safegrd), which copies the config and key from a Secret into an in-memory volume. [Where to run it](https://safegrd.dev/docs/deploy) has the steps for each. If you have no server, the [GitHub Action](https://github.com/safegrd/backup-action) backs up from a scheduled workflow and drills into a database it creates on the runner. [Back up Supabase with GitHub Actions](https://safegrd.dev/guides/supabase-backup-github-actions) walks through it. If your plan includes it, SafeGrd can also take the backup itself. Open _Back up on SafeGrd_ under Surfaces in the console and give it the connection string, or a bucket's S3 key for Supabase Storage. A machine SafeGrd starts for each backup does the rest ([how](https://safegrd.dev/docs/hosted)). None of these enrolls the host for you. After installing, run `safegrd login` and then `safegrd enroll`. At run time, a database surface needs its engine's client tools on the host. [Databases](https://safegrd.dev/docs/surfaces/databases) lists them per engine. SQLite, files and mail need nothing else. The CLI is [source-available](https://github.com/safegrd/cli) (Business Source License 1.1). Building it needs only Go: `go build ./cmd/safegrd` from a clone is the whole build. Release archives and their checksums are on the [releases page](https://github.com/safegrd/cli/releases). ## Try it before you sign up safegrd try postgres://app:secret@db.example.com/app You need no account for this. `try` backs up the database, encrypted on your host, to SafeGrd's storage. It also takes a MySQL, MongoDB or SQLite URL, or a directory. Then it restores the backup in memory from storage, prints the tables and rows it read back, and prints a sign-up link. Sign up to keep the backup and see its drill in the console. The connection string stays on your host, in a file beside the config that only you can read. To keep it in the environment instead, pass it as `env:DATABASE_URL`. The trial backup is kept for 24 hours and is not locked. SafeGrd keeps your key sealed and releases it only to this host, so you can restore even after losing the host. For a customer-managed key, pass `--key-custody local`. You can run `safegrd list`, `verify` and `restore` from the same host straight away. safegrd signup Opens the sign-up page with a code to check against the one in the terminal. When you sign up there, the host, its backup and its daily schedule become your new account's organization. You do not need to run anything more on the host. Verify your address from the email to lift the trial's limits. From the next backup on, every backup is locked under your plan. If nobody signs up, the backup and its key are deleted after 24 hours. safegrd login, then safegrd enroll The trial is for a new account. If you already have an organization, run these two commands on the host instead. The host enrolls in your organization as a new host, with the same surfaces and the same key, managed the way you chose at `try`. The trial and its backup stay behind and are deleted at the trial's deadline. You can run one trial per machine at a time and a few per day, each up to 500 MB. ## Two ways to set a host up Whichever of these you run first generates the Age keypair. The other finds the keypair already there, so the order does not matter. ### safegrd login, then safegrd enroll: connected to SafeGrd safegrd login safegrd enroll This is what the installer runs for you. `login` prints a link and a code that you approve in any browser. `enroll` then registers this host, so the console can show its snapshots, schedule Fire Drills against them and tell you when the host goes quiet. If this machine has no config yet, `enroll` runs the local setup first. If no browser can be involved at all, pass a token from [Tokens](https://safegrd.dev/dashboard#keys) in the console instead: `safegrd enroll --token sg_pat_YOUR_TOKEN`. ### safegrd init: standalone and offline safegrd init --storage s3 --s3-bucket my-worm-bucket `init` runs only the part on the host. It generates the keypair and writes `~/.safegrd/config.yaml`. It does not contact a SafeGrd server, though with an S3 bucket it asks the bucket whether Object Lock is on. A host set up this way backs up, restores and verifies on its own. Use it on an air-gapped host, or to try SafeGrd without an account. The key `init` generates stays on the host. If you enroll the host later, it keeps a customer-managed key. For a SafeGrd-managed key, enroll without running `init` first. ## Who holds the private key You choose how each host's key is managed when the host enrolls, with `--key-custody`. The choice decides who can decrypt the host's snapshots, and whether you can still restore them after losing the host. SafeGrd keeps the key unless you pass `local`. This applies to a key the host generates. A key you supply yourself is never sent anywhere. Each surface in the console shows how its key is managed. | Mode | What happens | | --- | --- | | --key-custody safegrd (default) | **SafeGrd-managed key.** SafeGrd keeps your key sealed and releases it only to your organization's enrolled hosts, for restores and drills. If you lose a host, you still have your backups: enroll a new host and restore ([the steps](https://safegrd.dev/docs/runbooks#managed)). | | --key-custody local | **Customer-managed key.** Only you can decrypt these backups. The private key is written to `key_path` at 0600, and only the public half is registered. Keep a copy of the key file somewhere safe. | ### With a customer-managed key, keep a copy of ~/.safegrd/keys/daemon.key. It is the key that decrypts your snapshots, and it stays with you. Store a copy with your most important credentials before the first backup. The private key is never written to `config.yaml`. It lives at `key_path`. When a restore or a drill needs it, the CLI reads it from there or from `SAFEGRD_PRIVATE_KEY`. [Security and key custody](https://safegrd.dev/docs/security) lists what SafeGrd holds and what it does not. ## By platform Each of these pages covers one platform: the hostname, role and URL to connect with, how to restore, and how to check the backup: [Supabase](https://safegrd.dev/supabase-backup), [Neon](https://safegrd.dev/neon-backup), [Amazon RDS](https://safegrd.dev/rds-postgres-backup), [Railway](https://safegrd.dev/railway-backup), [Render](https://safegrd.dev/render-backup), [Fly.io](https://safegrd.dev/fly-postgres-backup), [DigitalOcean](https://safegrd.dev/digitalocean-postgres-backup), [Heroku](https://safegrd.dev/heroku-postgres-backup), [a Hetzner VPS](https://safegrd.dev/hetzner-postgres-backup) and [Coolify](https://safegrd.dev/coolify-backup). For Docker Compose and Kubernetes, see [Where to run it](https://safegrd.dev/docs/deploy). ## Where configuration lives | Path | What it is | | --- | --- | | ~/.safegrd/config.yaml | The host's config: surfaces, storage sink, retention and encryption public key. Keep it at 0600, because it holds connection strings. | | ~/.safegrd/keys/daemon.key | The Age private identity, at 0600. With a customer-managed key, it never leaves the host. | | ~/.safegrd/ | Daemon state and per-surface locks, unless `state_dir` says otherwise. | To point a command at a different config file, use the global `--config` flag. To point it at a different SafeGrd server, use `--server-url` or `SAFEGRD_SERVER_URL`. Run `safegrd config validate` to check a file before the daemon uses it. ## Check the host ``` # permissions, keys, sink reachability, Object Lock, clock skew, surfaces safegrd doctor ``` Run `safegrd doctor` after setup and after every config edit. It checks: - the config file's permissions - that the keypair is present and coherent - that the storage sink answers and has Object Lock turned on - that SafeGrd's server answers - that the host clock has not drifted It also opens every configured surface with the credential its backup will use, including a credential SafeGrd holds. On an enrolled host, it sends the results to the console, which shows the host as checked or says what failed. For CI, `--json` makes the output machine-readable. --- Source: https://safegrd.dev/docs/deploy # Where to run it Run the daemon anywhere that can reach what it backs up. Every option below runs the same `safegrd daemon` with the same config. To move from one to another, copy `~/.safegrd`. | Where | Use it when | | --- | --- | | [A service on the host](https://safegrd.dev/docs/deploy#service) | The database or files are on a VM or a server you can SSH into. | | [Docker Compose](https://safegrd.dev/docs/deploy#compose) | The database runs in Compose, or you want the client tools without installing them on the host. | | [Kubernetes CronJob](https://safegrd.dev/docs/deploy#cronjob) | You run on Kubernetes and an hourly pass is often enough. | | [Kubernetes with Helm](https://safegrd.dev/docs/deploy#helm) | You run on Kubernetes and want the daemon resident, so a "Back up now" or a drill from the console starts within a minute. | | [GitHub Actions](https://safegrd.dev/docs/deploy#actions) | You have no server to run it on, and a scheduled workflow can reach the database. | | [On SafeGrd](https://safegrd.dev/docs/deploy#on-safegrd) | You do not want to run it on anything of yours. SafeGrd starts a machine for each backup, if your plan includes it. | Whichever you pick, enroll the host once first ([Install and enroll](https://safegrd.dev/docs/install)). Enrolling writes `~/.safegrd/config.yaml`, and with a customer-managed key it writes the key file next to it. In a container, keep that directory in a volume or a Secret. ## A service on the host Run `safegrd daemon install`. It writes a systemd unit on Linux or a launchd agent on macOS, and starts it. [Daemon](https://safegrd.dev/docs/daemon) covers the unit, its environment and its logs. ## Docker Compose Use the image `ghcr.io/safegrd/cli`. It includes `pg_dump` 18, the MariaDB client, the MongoDB tools and a PostgreSQL 18 server for sandbox drills, and it runs as its own user. The example [compose.yaml](https://github.com/safegrd/cli/blob/main/packaging/compose/compose.yaml) runs it next to a PostgreSQL service and keeps its config and key in the `safegrd-home` volume. ``` # 1. Enroll once: writes the config and key into the volume docker compose run --rm safegrd enroll --token sg_pat_... --claim # 2. Start the daemon docker compose up -d # 3. Check it docker compose exec safegrd safegrd doctor docker compose exec safegrd safegrd list ``` Use the claim code the console shows when you add a host. Enrolling with it writes the surfaces into the config. If a surface's credential is `{from: env, name: APP_DATABASE_URL}`, it reads its connection string from the service's environment, and the password stays out of the config. Size the volume for the largest database you back up, because a sandbox drill restores into a throwaway cluster there. With a customer-managed key, keep a copy of the key somewhere safe outside the volume. ## Kubernetes CronJob [cronjob.yaml](https://github.com/safegrd/cli/blob/main/packaging/kubernetes/cronjob.yaml) runs `safegrd daemon run --once` every hour. Each pass backs up every surface that is due, runs a drill if SafeGrd's server asks for one, and exits. Enroll on any machine, then put the config and the key in a Secret: ``` kubectl create secret generic safegrd \ --from-file=config.yaml=$HOME/.safegrd/config.yaml \ --from-file=daemon.key=$HOME/.safegrd/keys/daemon.key kubectl apply -f cronjob.yaml ``` With a SafeGrd-managed key, leave `daemon.key` out. The CLI refuses a config or key file that other users can read. A Secret volume cannot be both 0600 and owned by the container's user, so an init container copies both files into an in-memory volume at 0600. The daemon keeps its state (the last backup of each surface, the weekly and monthly slots) on a small PersistentVolumeClaim, so a restarted pod does not take an extra backup. `concurrencyPolicy: Forbid` stops two passes from backing up the same surface at once. A "Back up now" or "Drill now" from the console waits for the next pass. To wait less than an hour, change `schedule` so the CronJob runs more often. To have them start within a minute, run the daemon resident with the Helm chart below. ## Kubernetes with Helm The [Helm chart](https://github.com/safegrd/cli/tree/main/packaging/helm/safegrd) runs `safegrd daemon run` as a Deployment and reads the same Secret as the CronJob. Install it from a clone of [safegrd/cli](https://github.com/safegrd/cli): ``` helm install safegrd ./packaging/helm/safegrd ``` Set `existingSecret` to the Secret's name (`safegrd` by default). Use `env` to add the variables your surfaces read their credentials from. A sandbox drill restores into a throwaway cluster on the state volume (`persistence.size`, 2 Gi by default). Size it for your largest database plus a quarter. If a drill would leave less than 5% of the volume free, it runs in memory instead, and the console says so. ## GitHub Actions The [GitHub Action](https://github.com/safegrd/backup-action) backs up from a scheduled workflow. Its drills restore into a database it creates on the runner. For a walkthrough, see [Back up Supabase with GitHub Actions](https://safegrd.dev/guides/supabase-backup-github-actions). The action backs up PostgreSQL, MySQL, MariaDB, MongoDB and SQLite from a connection string. It also backs up a folder or a mailbox that your config names as a surface. Its [README](https://github.com/safegrd/backup-action#readme) has a workflow for each. Each job on a GitHub runner starts empty, so use `format: tar` when you back up a folder there. ## On SafeGrd Under Surfaces in the console, click _Back up on SafeGrd_ and give it a connection string. For each backup, SafeGrd starts a machine that runs it. SafeGrd keeps the connection string encrypted and uses it only on that machine. [How it works](https://safegrd.dev/docs/hosted). --- Source: https://safegrd.dev/docs/cli # Command reference Every `safegrd` command and its flags. Run `safegrd --help` on the host for the same list. If this page and the binary disagree, trust the binary. ## A first run, start to finish Four commands take a host from nothing to a restored database. The windows below show real output from one run on a host called `prod-app-01`. It backs up a PostgreSQL database named `shop` to SafeGrd-hosted storage. Each window plays when you scroll to it. Click one to see the whole run at once. ### 1. Enroll the host Get the claim code from _Add surface_ in the console. It tells the host which surfaces to back up. Get the token from _Tokens_. If you leave out `--token`, you approve a login in any browser instead. ### 2. Take the first backup One pass of the daemon backs up every surface that is due. A new surface is due straight away. After that, `safegrd daemon install` keeps the daemon running. ### 3. Check that it restores A Fire Drill loads the snapshot into an empty database and checks every table against what the backup recorded. The daemon runs drills on your plan's schedule. Here the same drill runs by hand. ### 4. Restore it into an empty database The snapshot comes down still encrypted, and this host decrypts it. Setup [try](https://safegrd.dev/docs/cli#try)[signup](https://safegrd.dev/docs/cli#signup)[init](https://safegrd.dev/docs/cli#init)[enroll](https://safegrd.dev/docs/cli#enroll)[keygen](https://safegrd.dev/docs/cli#keygen)[claim](https://safegrd.dev/docs/cli#claim)[login](https://safegrd.dev/docs/cli#login)[whoami](https://safegrd.dev/docs/cli#whoami) Protect [backup](https://safegrd.dev/docs/cli#backup)[guard](https://safegrd.dev/docs/cli#guard)[wal](https://safegrd.dev/docs/cli#wal) Recover and prove [restore](https://safegrd.dev/docs/cli#restore)[verify](https://safegrd.dev/docs/cli#verify)[find](https://safegrd.dev/docs/cli#find)[check](https://safegrd.dev/docs/cli#check)[verify history](https://safegrd.dev/docs/cli#verify-history) Storage [list](https://safegrd.dev/docs/cli#list)[undelete](https://safegrd.dev/docs/cli#undelete)[export](https://safegrd.dev/docs/cli#export)[estimate](https://safegrd.dev/docs/cli#estimate)[keep](https://safegrd.dev/docs/cli#keep)[prune](https://safegrd.dev/docs/cli#prune) Unattended [daemon run](https://safegrd.dev/docs/cli#daemon-run)[daemon status](https://safegrd.dev/docs/cli#daemon-status)[daemon install](https://safegrd.dev/docs/cli#daemon-install) Diagnose [doctor](https://safegrd.dev/docs/cli#doctor)[config validate](https://safegrd.dev/docs/cli#config-validate)[mcp](https://safegrd.dev/docs/cli#mcp)[org](https://safegrd.dev/docs/cli#org)[project](https://safegrd.dev/docs/cli#project)[version](https://safegrd.dev/docs/cli#version) ## Global flags | Flag | Meaning | | --- | --- | | --config | Config file. Default `~/.safegrd/config.yaml`. | | --server-url | SafeGrd's server. Default `$SAFEGRD_SERVER_URL` if set, else `https://safegrd.dev`. | | -v, --version | Print the version and exit. | ## Exit codes A command that fails exits non-zero. When the CLI can tell which side failed, the exit code says so. A script can then retry when a service is down, and stop when the mistake is its own. | Code | Meaning | | --- | --- | | 0 | The command did what it was asked. | | 1 | Failed, for a reason not listed below. The `Error:` line says why. | | 3 | `guard` only: no locked snapshot was taken, so guard did not run the command. | | 10 | The source failed: the database, mailbox or directory refused the connection or the credential, or does not exist. | | 11 | SafeGrd's server failed: the CLI could not reach it, it answered with a server error, or it sent a reply this CLI cannot read (upgrade the CLI). | | 12 | Storage failed: the CLI could not open, write or read the bucket or directory the backups go to. | | 130 | Interrupted with Ctrl-C or SIGTERM. | | 140 | `wal fetch` could not read a WAL file that the archive may hold, for example because the storage did not answer or there was no key. PostgreSQL then stops recovery instead of ending the replay at that file. Fix the cause and start recovery again. A file the archive does not hold exits 1. | ## Setup ### try safegrd try [flags] Backs up one database or directory before you have an account. It then restores the backup in memory from SafeGrd's storage to check it, and prints the tables and rows it read back. It takes a PostgreSQL, MySQL, MongoDB or SQLite URL, or a directory. The URL stays on this host. Pass it as `env:VAR` or `file:/path` to keep the password out of the config. SafeGrd keeps the backup for 24 hours, unlocked. `safegrd try status` tells you whether it is still kept, signed up or gone. The [install guide](https://safegrd.dev/docs/install#try) covers the limits and what happens next. | Flag | Meaning | | --- | --- | | --key-custody | `safegrd` (default) is a SafeGrd-managed key. SafeGrd keeps your key sealed and releases it only to this host, so you can restore even after losing the host. `local` is a customer-managed key. Only you can decrypt this backup. | | --name | This host's name in the console. Default: the hostname. | | --no-browser | Print the sign-up link without opening a browser. | ### signup safegrd signup [--no-browser] Opens the sign-up page for the backup that `try` took, with a code to match against the one in the terminal. Sign up there, or sign in to an account that has no organization yet. The host, its backup and its daily schedule then become your organization, and you run nothing more on the host. If your account already has an organization, run `safegrd login` and then `safegrd enroll` on the host instead. The host joins that organization as a new host, with the same surfaces and key. ### init safegrd init [flags] Writes the local config, and generates an Age keypair if this host has none. It does not contact SafeGrd's server. With `--storage s3`, it asks the bucket whether Object Lock is on. | Flag | Meaning | | --- | --- | | --node-name | This host's name in the console. Default: the hostname. | | --database-url | The database to back up, written to the config as its first surface: PostgreSQL, MySQL, MongoDB or SQLite. Pass `env:VAR` or `file:/path` to keep the URL out of the file. `safegrd backup` then backs it up. | | --laravel | Add a Laravel app as a surface, given the app's root (where `artisan` and `.env` are). See [Laravel](https://safegrd.dev/docs/surfaces/laravel). If the host already has a config, only the surface is added, and the key, the host's registration and the storage stay as they are. With `--laravel`, `--database-url` replaces the connection named in `.env`. | | --surface-id | With `--laravel`: the surface's id. Default: `APP_NAME` as a slug, or the directory's name if `APP_NAME` is not set. | | --migrations-table | With `--laravel`: the app's migrations table, if it is not named `migrations`. | | --storage | `local`, `s3` or `hosted`. Default `local`. | | --local-path | The directory backups go to with `--storage local`. | | --s3-bucket | Bucket for WORM storage. | | --s3-prefix | Key prefix in the bucket. Default `safegrd/snapshots`. | | --s3-region | Bucket region. Default `us-east-1`. | | --s3-endpoint | Custom endpoint, for MinIO, R2 or Wasabi. | | --retention-days | WORM retention in days. Default 14. | | --worm-mode | Object Lock mode for `--storage s3`: `COMPLIANCE` (default), `GOVERNANCE`, or `NONE` for a bucket with no Object Lock, such as DigitalOcean Spaces. | | --force | Overwrite an existing config. This discards its settings and generates a **new** keypair. Snapshots taken before that can only be read with the old key, so back up the old key first. | ### enroll safegrd enroll [flags] Registers this host with SafeGrd's server, and runs local setup first if the host has none. A personal access token enrolls new hosts only. A host that is already enrolled keeps its node token. If you lost the token or rebuilt the host, click _Rotate token_ on the host under Hosts in the console. Then run `safegrd enroll --token --node-id ` on the host. The host goes on backing up to its own key. With a SafeGrd-managed key, enroll fetches the key and writes it to `key_path`. With a customer-managed key, put the key file at `~/.safegrd/keys/daemon.key` first. | Flag | Meaning | | --- | --- | | --token | A personal access token to register a new host, or a node token together with `--node-id`. Takes `env:VAR` or `file:/path`. | | --node-id | The id of the host the node token belongs to. The console shows it beside the token. Leave it out with a personal access token. | | --node-name | This host's name in the console. Default: the hostname. | | --org | The organization to enroll into. Needed only when the credential can see more than one. | | --project | Project id or slug. Defaults to the organization's default project. | | --key-custody | `safegrd` (default) stores a SafeGrd-managed key, sealed, on SafeGrd's server. `local` creates a customer-managed key on this host and sends only the public half. Ignored if you supply your own key. | | --storage | Where this host's backups go if its project has no bucket. `hosted` uses SafeGrd-hosted storage, if your plan includes it. `local` keeps them in a directory on this host. | | --allow-unconfigured | Enroll a host that has nowhere to back up to yet, with no storage in its config or its project. Without this flag, enroll stops and tells you what to set, so the host does not enroll and then fail its first backup. | ### keygen safegrd keygen [--out path] | safegrd keygen --recipient path | safegrd keygen --held [--snapshot id] --out path Generates an age key pair for a customer-managed key. You need no other tools. keygen writes the private key at mode 0600 and does not print it. It prints the public key. Put that in `encryption.public_key` on the hosts that back up, and leave `key_path` empty. Those hosts can then encrypt but cannot decrypt. Keep a copy of the private key somewhere safe, such as a password manager. keygen does not overwrite an existing file. On an enrolled host, `--held` writes a copy of a SafeGrd-managed key to `--out` instead. By default that is this host's key. With `--snapshot` it is the key that snapshot was sealed to, for example a snapshot of a database SafeGrd backs up for you, or of a host you lost. SafeGrd keeps the key as before, and the key access record in **Settings** lists the copy. | Flag | Meaning | | --- | --- | | --out | Where to write the private key. Default: `~/.safegrd/keys/daemon.key`. Required with `--held`. | | --recipient | Print the public key for the private key in this file. Generates nothing. | | --held | Write a copy of the SafeGrd-managed key. Generates nothing. | | --snapshot | With `--held`: write the key this snapshot was sealed to, instead of this host's. | ### claim safegrd claim Adds the surfaces you named for this host in the console (**Add surface** on the host's row under Surfaces) to an enrolled host's config. Nothing changes on the host until you run it, and it only adds. It leaves any surface the config already has as it is, and keeps the previous config as `config.yaml.bak`. Running it again adds nothing. A running daemon reads the new config at the start of its next pass. On a new host, run `safegrd enroll --claim ` instead. ### login safegrd login [flags] Signs this machine in to your SafeGrd account. It prints a link and a code, which you approve in a browser on any device, so it works the same over SSH. On an enrolled host, login refuses to run while SafeGrd still accepts the host's node token, because it would replace that token. On a host where nobody signed up for the `try` backup, login replaces the trial's token. `enroll` then registers the host as a new host in your organization. | Flag | Meaning | | --- | --- | | --token | Personal access token (`sg_pat_...`) for CI and headless use. | | --no-browser | Print the URL instead of opening a browser. | ### whoami safegrd whoami Prints who you are signed in as and the active server session. ## Protect ### backup safegrd backup [flags] Streams a surface, compresses it with zstandard, encrypts it to your Age public key and writes the ciphertext to WORM storage. Unencrypted data never touches disk. The one exception is a SQLite surface's private temporary copy on the same host ([why](https://safegrd.dev/docs/surfaces/sqlite)). | Flag | Meaning | | --- | --- | | --database-url | Database connection string. The scheme picks the surface: `postgres://…` for PostgreSQL, `mysql://…` (or `mariadb://…`) for MySQL, `mongodb://…` for MongoDB, `sqlite:///path/to/file.db` for SQLite. | | --surface | The id of one surface under `surfaces:` in the config. Backs it up now, whatever its schedule, the way the daemon would, with the surface's own storage, retention and credential. Run it before a migration. Do not combine it with the flags that describe a surface. | | --files | Directory tree to back up, or a bucket as `s3://bucket/prefix`. Selects the files surface. | | --source-endpoint, --source-region, --source-access-key, --source-secret-key, --source-session-token | With `--files s3://…`: where the bucket is, and the keys that read it, as `env:VAR` or `file:/path`. Add the session token when the key is temporary. `--files s3://*` reads every bucket the key lists, each as a directory in the archive. See [a bucket as a surface](https://safegrd.dev/docs/surfaces/files#bucket). | | --email | Back up an IMAP mailbox. Selects the email surface. | | --exclude | Glob patterns to leave out of a file backup, for example `'*.tmp,node_modules/*'`. | | --note | A note on the snapshot, `key=value`, repeatable. See [note](https://safegrd.dev/docs/cli#note). | | --compression | How hard to compress: `zstd:1` (fastest) to `zstd:19` (smallest). A restore needs no matching setting. In the config, set `compression` on the surface. | | --format | How `--files` or a database is stored. `repo` (the default) is incremental: each run uploads only what changed. `tar` writes one archive per backup. See [incremental file backups](https://safegrd.dev/docs/surfaces/files#incremental) and [databases](https://safegrd.dev/docs/surfaces/postgres#incremental). | | --change-log | For a PostgreSQL backup in `repo` format: skip reading tables that nothing has written to since the last run. A trigger it puts on each table records the writes. See [the change log](https://safegrd.dev/docs/surfaces/postgres#change-log). | | --roles-without-passwords | PostgreSQL: leave role password hashes out of the backup. By default, the roles the schema names are backed up with their password hashes, encrypted like everything else. See [roles, owners and grants](https://safegrd.dev/docs/surfaces/postgres#roles). | | --new-epoch | Start a new monthly epoch now, which uploads everything once. Works with `--format repo` or `--surface`. | | --rescan | Read every file, not only those whose size or times changed. Works with `--format repo` or `--surface`. | | --one-filesystem | Stay on the root's filesystem. On by default when the root is `/`. | | --email-host | IMAP hostname. | | --email-port | IMAP TLS port. Default 993. | | --email-user | Mailbox username or address. | | --email-password | Mailbox password. Use `SAFEGRD_EMAIL_PASSWORD` instead where you can, because a flag shows in the process table. | | --email-folders | Folders to back up. Default: all except spam and trash. | | --email-ca-file | A PEM bundle of extra CAs to trust for the IMAP server. They are added to the system roots and do not replace them. Also `SAFEGRD_EMAIL_CA_FILE`. | | --retention-days | How many days this backup is locked. If you pass no `--keep-*` flag with it, the config's `defaults.keep_*` tiers do not apply to this run. | | --keep-daily, --keep-weekly, --keep-monthly | Lock the first one-off backup of each UTC day, ISO week or month for this many days, weeks or months ([GFS](https://safegrd.dev/docs/storage#gfs)). Default: the config's `defaults.keep_*`. | | --s3-bucket | Write to this bucket instead of the project's or the config's. | | --s3-prefix | Key prefix in the bucket. | | --s3-endpoint | S3 endpoint, for MinIO, R2 or another S3-compatible store. | | --s3-region | S3 region. | | --s3-access-key | S3 access key id. | | --s3-secret-key | S3 secret access key, as `env:VAR` or `file:/path`. | | --tag | A prefix for the snapshot tag. | | --json | Print the result as JSON. | ### guard safegrd guard [--surface ID] [-- command [args...]] Backs up one surface, waits until the snapshot is uploaded and locked, then runs the command. If the backup fails or the snapshot is not locked, the command does not run and guard exits 3. See [Agent guard](https://safegrd.dev/docs/agents). | Flag | Meaning | | --- | --- | | --surface | The surface to back up, by its id in the config. | | --allow-unlocked | Run the command even when the backup is not locked, as on local storage or with `worm_mode: NONE`. | | --list | Print the rules that mark a command as destructive. | | --matches | Check a command against the rules and take no backup: exit 0 if it matches, 1 if not. | | --hook | Answer a pre-command hook on stdin: `claude-code`, `cursor` or `codex`. | ### wal safegrd wal push | base | bundle | status | point | restore | drill | fetch Point-in-time recovery for a PostgreSQL, SQLite, MySQL or MongoDB surface with a `wal:` block. `wal restore --to ` restores to any second in the recovery window. Run `wal point` before a risky change to mark a restore point. For PostgreSQL, set `archive_command` to `wal push`. It compresses and encrypts each WAL file on the host and writes it to the surface's storage. The daemon runs `wal base` weekly and `wal bundle` daily. The bundle locks the day's WAL in one object. For MySQL and MongoDB, the daemon follows the binary log or oplog, and `wal restore --target-url` loads a base backup into an empty database and replays the log up to the moment you ask for. Setup and the recovery window are on the [PostgreSQL](https://safegrd.dev/docs/surfaces/postgres#pitr), [SQLite](https://safegrd.dev/docs/surfaces/sqlite#pitr), [MySQL](https://safegrd.dev/docs/surfaces/mysql#pitr) and [MongoDB](https://safegrd.dev/docs/surfaces/mongodb#pitr) pages. ## Recover and prove ### restore safegrd restore --snapshot | --path | --table | --from [flags] Downloads a snapshot still encrypted, decrypts it on this machine and restores it into a database or a directory. With `--from`, it reads a copy that `safegrd export --to-dir` made instead of the storage in the config ([Runbook 4](https://safegrd.dev/docs/runbooks#from-export)). | Flag | Meaning | | --- | --- | | --snapshot | The snapshot id to restore. Leave it out to restore the newest kept version of one `--path` or one `--table`, or when `--from` names a `.safegrd` file. | | --from | The export to restore from: the directory `export --to-dir` wrote (add `--snapshot`), or one `.safegrd` file in it. | | --from-bucket | Read the snapshot from one of the surface's other buckets, by its name in `copies:` or in the console ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). | | --request | Run a restore someone requested in the console, by its id (`rst_...`). The request sets the snapshot, the target and what to restore, and the CLI reports the outcome back. The only other flags it takes are `--key-path` and `--private-key` ([Restore from the console](https://safegrd.dev/docs/verify#console)). | | --target | The database URL to restore into, `postgres://…` or `mysql://…` to match the snapshot. The database must be empty. For a files snapshot, `s3://bucket/prefix` uploads each file into that bucket with the content type it had at backup time ([restore into a bucket](https://safegrd.dev/docs/surfaces/files#bucket-restore)). | | --source-endpoint, --source-region, --source-access-key, --source-secret-key, --source-session-token | With `--target s3://…`: where the bucket is, and the keys that write to it, as `env:VAR` or `file:/path`. For a snapshot of every bucket, `--target s3://*` puts each file back in the bucket it came from. | | --target-dir | The directory to restore file and email snapshots into. For a WordPress site or a Laravel app, use it together with `--target`: the database goes to `--target` and the files go here. | | --path | For an incremental files snapshot: restore only this path, relative to `/`. Repeatable. `*`, `?` and `**` work as wildcards. | | --table | Restore this table alone into `--target`, which may hold other tables. Repeatable. For PostgreSQL, from any backup, give `schema.table`, with either part quoted the way `psql` prints it. For MySQL, MariaDB and SQLite, from an incremental backup, give the table's name. For MongoDB, from an incremental backup, give one collection's name. The table loads into an empty table of the same definition. If the target does not have the table, the restore creates it as the backup defined it, with its indexes (and for PostgreSQL its schema, sequences and constraints). The target's other tables are left as they are. A full (`tar`) backup of MySQL, SQLite or MongoDB restores whole. | | --as | With one `--table` of a PostgreSQL backup: restore it beside the live table under this name (`schema.table_restored`). The live table stays as it is. The copy has the live table's columns and NOT NULL rules, and no indexes, keys, checks or defaults. Compare the two with SQL and copy back what you need. The live table must still have the definition the backup recorded. | | --rows | With one `--table`: restore only the rows that match a SQL condition, such as `"id IN (4211, 4212)"` or `"deleted_at > '2026-10-30'"`, into the table as the target has it. The restore loads the backup's rows into a temporary table inside the target's own transaction, then writes the matching rows into the table by primary key. The target's other rows and tables stay as they are. A row the target already has is left alone and counted. The table must exist in the target with the definition the backup recorded, and it must have a primary key. The command prints how many rows it inserted, replaced and left. It does not move a sequence the table owns. If you put back a row with an id past the sequence, the next insert collides with it until you move the sequence with `setval`. | | --overwrite | With `--rows`: replace a row the target already has with the backup's. Without this flag, a restore does not write over existing data. | | --version | With one `--path` or `--table`: restore this version, numbered as `safegrd find` lists them, instead of naming a `--snapshot`. | | --surface | With `--version`: the surface whose repository holds the file, when more than one does. | | --schema | PostgreSQL only: restore this schema's objects and rows into a database that already has other schemas, such as a Supabase project. Repeatable. The target must not already have the schema's objects. | | --data-only-schema | PostgreSQL only: load this schema's rows into the tables the target already has, and create nothing. If the target lacks a table, or the restoring role cannot insert into it, the restore leaves it out and names it with its row count. If none of a schema's tables can be loaded, the restore refuses that schema. Repeatable, and you can combine it with `--schema`. | | --salvage | For an incremental files snapshot with a damaged or missing pack: restore every file that still matches its SHA-256, leave the rest out of the target, and list each one with the reason. Exits non-zero if any file is left out. Without this flag, such a restore writes nothing and tells you to run it again with `--salvage`. | | --without-erasures | PostgreSQL only: restore even when the surface's erasure record cannot be read. Rows erased after the backup may come back, and the restore says so. See [the erasure record](https://safegrd.dev/docs/surfaces/postgres#erasures). | | --keep-cron-active | PostgreSQL only: create the snapshot's pg_cron jobs as they were at backup time. By default, each job is created turned off. See [pg_cron jobs](https://safegrd.dev/docs/surfaces/postgres#pg-cron). | | --parallel | For incremental PostgreSQL backups: how many connections load tables at once. Default 4, capped by the server's free connections. With more than one, the restore is not one transaction, so if it fails, drop the target and restore again. `1` loads everything in one transaction. | | --no-owner | PostgreSQL only: restore without owners, grants and default privileges, so every object belongs to the restoring role. The roles the schema names are still created, because a policy cannot be restored without them. See [roles, owners and grants](https://safegrd.dev/docs/surfaces/postgres#roles). | | --to-sql | Write a PostgreSQL snapshot into this new directory as files `psql` can load: the schema as SQL, each table's rows as binary `COPY`, and `load.sql`. Run `psql "$URL" -f load.sql` from that directory. The files are not encrypted. The directory is created with mode 0700. | | --key-path | Path to the Age private identity file. | | --private-key | The Age private identity inline (`AGE-SECRET-KEY-1...`). | ### verify safegrd verify --snapshot [flags] Runs a Fire Drill: downloads the snapshot, decrypts it and checks what came back. By default it restores in memory, so you need no database. | Flag | Meaning | | --- | --- | | --snapshot | Snapshot id to verify. Required. | | --surface | For a surface with `watch:`, drill the backup it last read, in the place the other tool put it, instead of a snapshot. The drill runs in memory or loads into `--sandbox-target`. `safegrd backup --surface` runs that surface's check. See [watched backups](https://safegrd.dev/docs/surfaces/postgres#watch). | | --in-memory | Restore in memory, with no target database, even when `--sandbox-target` is set. This is the default when `--sandbox-target` is not set. | | --sandbox-target | An empty database to restore into for a full drill. `verify` does not empty it afterwards. If the drill fails, verify prints where the sandbox is. | | --keep-sandbox | When a sandbox drill fails, record that the sandbox is kept for someone to inspect. | | --query | Read rows out of a database backup without restoring the table they came from. The CLI restores the snapshot into `--sandbox-target`, or into a PostgreSQL cluster the host starts for the run, and prints the query's rows as CSV. The run is not reported as a drill, and stdout carries only the CSV. Example: `safegrd verify --snapshot --query "SELECT * FROM orders WHERE id = 4211"`. To put rows back, use `restore --rows`. | | --out | With `--query`: write the CSV to this file instead of stdout. | | --keep | With `--query`: keep the PostgreSQL cluster the run started running for a day, and print how to connect to it with `psql`. | | --show-url | When a sandbox drill fails, print the sandbox URL with its password. | | --key-path | Path to the Age private identity file. | | --private-key | The Age private identity inline. | | --s3-bucket | Read the snapshot from this bucket instead of the configured storage. | | --s3-prefix | Key prefix in the bucket. | | --s3-endpoint | S3 endpoint, for MinIO, R2 or another S3-compatible store. | | --s3-region | S3 region. | | --from-bucket | Read the snapshot from one of the surface's other buckets, by its name ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). | | --s3-access-key | S3 access key id. | | --s3-secret-key | S3 secret access key, as `env:VAR` or `file:/path`. | ### note safegrd note key=value... [--json] Sets notes on a snapshot, such as `release=v2.3` or `ticket=OPS-12`. `key=` with no value removes that note. To set notes while the backup runs, use `backup --note key=value`. SafeGrd stores the notes outside the locked backup, so you can change them later. `safegrd list` shows them. Keys use lowercase letters, digits, `_`, `.` and `-`. A snapshot holds up to 20 notes. ### find safegrd find ... [flags] Lists every kept version of the matching files in incremental backups. For each version it shows when it was first and last seen, its size and SHA-256, and how many snapshots hold it. Patterns are paths relative to `/`. A directory matches everything below it. Run it on a host that has the private key. | Flag | Meaning | | --- | --- | | --surface | Search this surface only. Without it, the host searches the surfaces it has the key for and says how many others it skipped. If you name another host's surface and it has a SafeGrd-managed key, find fetches that key. | | --table | List the kept versions of a table instead. Give `schema.table` for PostgreSQL, and the name alone for a MySQL, MariaDB or SQLite table or a MongoDB collection. Repeatable. `restore --table --version ` restores one of them. | | --deleted | List only files the newest snapshot no longer holds. | | --from-bucket | Search one of the surface's other buckets, by its name, instead of this host's storage ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). | | --json | Print the histories as JSON. | | --key-path | Path to the Age private identity file. | | --private-key | The Age private identity inline. | ### check safegrd check --snapshot | --epoch | --all [--read-data] Checks incremental backups without restoring them. It confirms that every pack a snapshot names is in storage and agrees with its index, and that every directory listing decodes. It recomputes the content digest from them, compares it with the recorded one, and checks that the catalog agrees. If anything is wrong, check names it and exits non-zero. | Flag | Meaning | | --- | --- | | --read-data | Also open every chunk and check its SHA-256. This downloads the snapshot's data. | | --key-path | Path to the Age private identity file. | | --private-key | The Age private identity inline. | ### verify history safegrd verify history [flags] Checks a host's attestation chain, and each record's Ed25519 signature against the key the record names. | Flag | Meaning | | --- | --- | | --node | The node id of the host whose history to check. | | --file | Local JSON file of verifications, for an offline audit. | | --save | Also write the records fetched from SafeGrd to this file, for a later `--file` check. | | --key | One hex-encoded Ed25519 public key, checked against every record. `file:/path` reads it from a file. | | --keys-file | The JSON from `/api/v1/attestations/public-key`, saved while online. It holds the active key and the retired ones, for a chain that spans a key rotation. | | --json | Print the result as JSON, with the keys that signed the chain. | ## Storage ### list safegrd list Lists every snapshot in the bucket, with the host it was filed under, what it holds and its retention state. `restore` and `verify` find a snapshot whichever host it is filed under, so when you rebuild a lost host you do not need the old node id ([Recovery runbooks](https://safegrd.dev/docs/runbooks)). Incremental file backups are listed in a second table. For each snapshot it shows the month (epoch) it belongs to, whether its run opened the month or followed on from it, the size of the tree it holds and the bytes its run uploaded. | Flag | Meaning | | --- | --- | | --json | Print a JSON array on stdout, one object per snapshot with `snapshot_id`, `created_at`, `status`, `locked` and `locked_until`. Every other line goes to stderr. | | --from-bucket | List one of the surface's other buckets, by its name, instead of this host's storage ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). | ### undelete safegrd undelete [flags] Brings back snapshots hidden by delete markers in a versioned bucket. It removes only delete markers, by version id, and cannot remove a version that holds data. With `--all`, it rolls the bucket back to before the deletion, every snapshot and sidecar, without uploading anything again ([how a deletion is undone](https://safegrd.dev/docs/storage#undelete)). | Flag | Meaning | | --- | --- | | --snapshot | Make one snapshot visible again. | | --all | Clear delete markers from every hidden snapshot. | ### export safegrd export --to-dir | --to-bucket [flags] Copies every snapshot and its metadata, still encrypted, to a directory or to a bucket you own. Use it to move off hosted storage or to keep an offline copy. It needs no key. `--snapshot` and `--node` narrow what it copies. An incremental files backup is copied a whole month (epoch) at a time, because a snapshot needs every object of its month. `--snapshot` copies the month that holds that snapshot. Each copy is checked against the digest recorded at backup time, and a snapshot keeps its lock in the destination bucket. A snapshot already in the destination is skipped, so you can run an interrupted export again. Credentials for `--to-bucket` come from the standard AWS environment. `safegrd restore --from` restores from a directory export. | Flag | Meaning | | --- | --- | | --to-dir | Export into this local directory. | | --to-bucket | Export into this S3 bucket. | | --to-endpoint | S3 endpoint of the bucket (MinIO, B2, R2, ...). | | --to-region | Region of the bucket. Default `us-east-1`. | | --to-prefix | Key prefix in the bucket. Default `safegrd/snapshots`. | | --to-worm-mode | `COMPLIANCE` (default), `GOVERNANCE` or `NONE`. | | --snapshot | Export only this snapshot. Repeat it or give a comma-separated list. An id that is not in storage is an error. | | --node | Export only the snapshots of the hosts with these node ids. Repeatable. | ### copies safegrd copies status | sync [--surface ] [--copy ] Keeps a surface's other buckets up to date ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). `status` shows, for each surface and copy, how many backups within the copy's retention the copy is missing. It exits non-zero when a copy is behind. `sync` writes the missing backups from the surface's own storage. It checks for an archive before writing it, so no archive is written twice, and it copies an incremental backup's run object by object. The daemon runs the same sync every half hour for any copy that is behind. | Flag | Meaning | | --- | --- | | --surface | Only this surface. | | --copy | Only this copy, by its name. | ### estimate safegrd estimate --database-url postgres://... [flags] Measures a PostgreSQL database and prints the monthly egress of weekly, daily and hourly backups at its provider, after the egress the provider plan includes. Every backup reads every row uncompressed, so one backup reads about the size of the tables without their indexes. The CLI reads the provider from the host name (Supabase, Neon, Amazon RDS). Its prices come from the providers' pricing pages, read on 5 October 2026. A warning on stderr names each schedule whose egress costs more a month than the SafeGrd plan you compare it with, at the price SafeGrd's server publishes. Your application's own traffic uses the same allowance, so pass what is left with `--included-gb`. For a database SafeGrd backs up for you, the console's add dialog prices the schedule before you add the surface (Price this schedule). | Flag | Meaning | | --- | --- | | --database-url | The database to measure. Default: the first PostgreSQL surface in the config. | | --surface | Measure this surface's database. | | --provider | `supabase`, `neon`, `rds` or `other`. Default: read from the host name. | | --provider-plan | The provider plan whose included egress applies: Supabase `free`, `pro` (default) or `team`; Neon `free`, `launch` (default) or `scale`. | | --egress-price-per-gb | The price in US dollars per GB, for `--provider other` or to replace the one built in. | | --included-gb | Free egress a month that your application does not already use. | | --schedule | Also price this schedule, written the way the config writes it. | | --plan | The SafeGrd plan to compare with: `starter`, `growth` (default) or `scale`. | | --json | Print the estimate as JSON. | ### keep safegrd keep --snapshot --until --yes Extends one snapshot's lock to the end of `--until` (UTC). The backup file, its metadata and its recovery document are extended together. Nobody can shorten a lock once it is extended, so the date can be at most 13 months from today and the command needs `--yes`. In your own bucket, the host extends the lock and tells SafeGrd's server. That needs `s3:PutObjectRetention` and `s3:GetObjectRetention`. On hosted storage, SafeGrd's server extends it. An incremental snapshot is kept as long as its month, and you cannot keep one on its own ([keep a snapshot longer](https://safegrd.dev/docs/storage#keep)). | Flag | Meaning | | --- | --- | | --snapshot | The snapshot to keep longer. | | --until | When the new lock ends: a date (locked to the end of that day, UTC) or an RFC 3339 time. | | --yes | Extend the lock. Without this flag, the command says what it would lock and stops. | ### prune safegrd prune [--dry-run] [--grace 24h] [--min-keep 3] [--from-bucket name] Deletes snapshots from your S3 bucket whose Object Lock retention expired more than a day ago, going by the bucket's own timestamp and retention status. It needs the `s3:DeleteObjectVersion` and `s3:GetObjectRetention` permissions. Set `storage.expire_after_lock: true` to prune automatically every day. prune always keeps: - The newest `--min-keep` snapshots of each surface (three by default), whatever their locks say. - The last known-good snapshot, while Threat Shield (SafeGrd's anomaly detection) reports an open anomaly. - The newest older backup whose drill passed, when none of the kept backups passed its Fire Drill and the oldest of them failed. prune says which backup it kept. Snapshot versions are removed before delete markers and metadata. An incremental files backup is pruned by month. A month's expired objects are deleted once every snapshot that could use them is past its retention, and never from the month that holds the surface's newest snapshot. Each of your other buckets is pruned next, by its own locks and `min_keep`, and prints its own `Prune offsite:` line. For a surface with `erasures:`, prune also prunes the erasure record. An object in it is deleted once its lock and the grace period have passed and no backup older than its newest erasure remains. The record's marker stays while the surface has a backup. On local storage, only the erasure record is pruned. | Flag | Meaning | | --- | --- | | --min-keep | How many of each surface's newest snapshots are never deleted, whatever their locks say. At least 1. Default 3. | | --dry-run | List what would be deleted, and delete nothing. | | --grace | How long after a lock ends a snapshot is kept anyway. Default `24h`. | | --from-bucket | Prune only this one of your other buckets, by its name ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). | ## Unattended ### daemon run safegrd daemon run [flags] Runs the daemon. On each pass it finds the surfaces that are due and backs them up, with a lock per surface so a surface never runs two backups at once. | Flag | Meaning | | --- | --- | | --once | Run one pass, then exit. Use it from cron or a Kubernetes Job. | | --interval | How often the daemon polls. Default 5m. With `--once`, set it to how often your scheduler runs the pass. The host then counts as a daemon at that interval. It picks up Back up now and Drill now from the console on its next run, and the console shows it as silent when the runs stop. | | --state-dir | Directory for state and locks. | ### daemon status safegrd daemon status [flags] Shows the current status of the daemon and of every surface in the config. | Flag | Meaning | | --- | --- | | --json | Print the status as JSON. | | --state-dir | State directory to read. | ### daemon install safegrd daemon install [flags] Installs the daemon as a systemd unit or launchd job, or prints the definition with `--print`. After that, use `daemon restart` and `daemon uninstall` to manage it. | Flag | Meaning | | --- | --- | | --user | Install as a user agent instead of a system service. | | --print | Print the service definition to stdout and install nothing. | ## Diagnose ### doctor safegrd doctor [flags] Checks what a backup on this host depends on: config permissions (0600 on POSIX), the keypair, that the WORM storage connects and that Object Lock works on it, that SafeGrd's server answers, and the host's clock skew. It also opens every surface with the credential its backup will use. It runs a query on each PostgreSQL, MySQL, MongoDB and SQLite database, logs in to each mailbox and opens each files root. For each database it finds a `pg_dump` or `mysqldump` that can dump that server. For each PostgreSQL surface it warns when the role can write, prints the read-only role to use, and names the tables where row-level security hides rows ([read-only role](https://safegrd.dev/docs/surfaces/postgres#role)). On an enrolled host, doctor reports the results to the console, which shows the host as checked. | Flag | Meaning | | --- | --- | | --json | Print the report as JSON. | | --agent-proof | Check instead whether an AI agent on this host could destroy the backups, and print the fix for each failure. See [Agent guard](https://safegrd.dev/docs/agents#agent-proof). | ### config validate safegrd config validate Checks the config's syntax, its file permissions and the surface definitions, without running anything. ### mcp safegrd mcp Runs an MCP server on stdin and stdout, so a coding agent on the same machine can use this host's safegrd (`claude mcp add safegrd -- safegrd mcp`). Its tools are status, list, doctor, backup, verify, restore and export, and each runs with this host's config. No tool deletes a backup or takes a private key. The restore tool writes only into a new or empty directory or an empty database. For the hosted MCP server, which needs no install, see the [MCP page](https://safegrd.dev/docs/mcp). ## Organization ### org safegrd org list | safegrd org members `org list` prints your organization's profile, plan and quota status. `org members` lists the team. `orgs` works as an alias. ### project safegrd project list [--org ] Lists the projects inside an organization. ### version safegrd version Prints the CLI version, commit and build date. Include it in any bug report. ## Environment variables | Variable | Meaning | | --- | --- | | SAFEGRD_SERVER_URL | SafeGrd's server, the same as `--server-url`. | | SAFEGRD_PRIVATE_KEY | The Age private identity, for restores and drills on a host without the key file. | | SAFEGRD_EMAIL_PASSWORD | Mailbox password. Use this instead of the flag. | | SAFEGRD_EMAIL_CA_FILE | PEM bundle of extra CAs to trust for IMAP. | ## Token lifetime A personal access token works until the date you pick when you create it, from 7 days to a year. The default is 90 days. The console shows the date beside each token, and does not offer tokens without an end date. Give each pipeline or cluster its own token, scoped to the one project it needs, and create the next one before the date. The token that `safegrd login` creates lasts 30 days. On a host, `enroll` replaces it with the host's own token. --- Source: https://safegrd.dev/docs/config # Config file reference Every key that `~/.safegrd/config.yaml` accepts is listed below. Commands read the file named by `--config`, or `~/.safegrd/config.yaml` when you leave it out. The file must be readable by its owner only (0600). A command refuses a file with wider permissions. `safegrd daemon install` writes the file's full path into the service definition, so the service finds it from any working directory. Run `safegrd config validate` to check the file's syntax and structure without running anything. Run `safegrd doctor` to check that credentials resolve and the storage sink is reachable. This page is checked against the CLI on every build. The build fails if a key exists and is not listed here, or if a key listed here does not exist. Here is a complete file for one database and one folder, and what `config validate` prints for it: ## server and node `safegrd init` and `safegrd enroll` write these. You rarely need to edit them. | Key | What it does | | --- | --- | | `server.url` | The address of SafeGrd's server. Default `https://safegrd.dev`. The CLI contacts it only when `server.token` is set. You can also set `SAFEGRD_SERVER_URL`. | | `server.token` | The host's token from `enroll` (`sg_tok_…`). Without a token the host runs standalone. It still backs up, prints that it is standalone, and reports nothing to SafeGrd. You can also set `SAFEGRD_SERVER_TOKEN`. | | `node.id` | The id the host reports under. Its snapshots go in the bucket under `//`. `enroll` sets it. | | `node.name` | The name `enroll` registers the host under. The console shows it. | | `node.project` | The project the host was enrolled into. If the project has a bucket set in the console, the host's backups go there. The CLI then refuses a `storage.bucket` that names a different bucket, and a `secret_access_key` when SafeGrd holds the bucket's key. If the project has no bucket, the bucket named here becomes the project's bucket. SafeGrd keeps a fingerprint of its key, not the key itself. | ## storage Where snapshots are written. A surface can replace the whole block with a `storage` block of its own. | Key | What it does | | --- | --- | | `storage.type` | `s3`, `local`, `sftp` ([an SFTP server](https://safegrd.dev/docs/storage#sftp)) or `hosted`. With `hosted`, snapshots go to SafeGrd's locked bucket through URLs that SafeGrd's server signs for each request ([Hosted storage](https://safegrd.dev/docs/storage#hosted)). Default `local`. | | `storage.bucket` | The S3 bucket. You can also set `SAFEGRD_STORAGE_BUCKET`. | | `storage.region` | The bucket's region. You can also set `SAFEGRD_S3_REGION`, which is read before `AWS_REGION`. | | `storage.endpoint` | An S3-compatible endpoint, such as MinIO, Backblaze B2, Wasabi or DigitalOcean Spaces. Leave it out on AWS. You can also set `SAFEGRD_S3_ENDPOINT`. | | `storage.prefix` | The key prefix inside the bucket. `init` writes `safegrd/snapshots`. To restore on another machine, use the same value there. | | `storage.node_id` | The folder under `prefix`. Defaults to `node.id`. Set it only to read another host's snapshots. | | `storage.retention_days` | How many days each snapshot is locked under Object Lock. It applies to every surface without a `retention_days` of its own. Default 14. A lock cannot be shortened once it is set, so choose the number with care ([Storage and retention](https://safegrd.dev/docs/storage)). | | `storage.expire_after_lock` | Off by default. Set it to `true` and, once a day, the daemon deletes the snapshots in your bucket whose Object Lock ended more than a day ago by the bucket's clock. It keeps each surface's newest `min_keep` snapshots, and the last known-good snapshot before an open anomaly. The bucket credentials need `s3:DeleteObjectVersion` and `s3:GetObjectRetention` ([Storage and retention](https://safegrd.dev/docs/storage)). | | `storage.min_keep` | How many of each surface's newest snapshots pruning keeps, whatever their locks say. Default 3. `safegrd prune --min-keep` sets it for one run. | | `storage.worm_mode` | `COMPLIANCE` (the default): nobody can delete a snapshot before its lock date. `GOVERNANCE`: a user or role with the bypass permission can. `NONE`: for a sink with no Object Lock. Write the value exactly as shown, in upper case. `sftp` storage accepts only `NONE`. | | `storage.storage_class` | The S3 storage class for each archive: `STANDARD` (the default), `STANDARD_IA` or `GLACIER_IR`. The manifest is always `STANDARD`. Each drill reads one copy and pays that class's retrieval price ([details](https://safegrd.dev/docs/storage#storage-class)). | | `storage.local_path` | The directory for `type: local`. Default `./safegrd-storage`. | | `storage.sftp.host` | The SSH server for `type: sftp` ([An SFTP server](https://safegrd.dev/docs/storage#sftp)). | | `storage.sftp.port` | Its SSH port. Default 22. | | `storage.sftp.user` | The account that writes the backups. | | `storage.sftp.path` | The directory on the server that backups go in, either absolute or relative to the account's login directory. To restore on another machine, use the same value there. | | `storage.sftp.password` | The account's password. Takes `env:VAR` or `file:/path`. | | `storage.sftp.private_key` | The account's private key, usually `file:/path/to/key`. The CLI cannot read a key that has a passphrase. | | `storage.sftp.known_hosts` | The full path to an OpenSSH `known_hosts` file that lists the server. The CLI refuses a server the file does not list, or lists with a different key. | | `storage.sftp.host_key` | Use this in place of `known_hosts`: the server's host key fingerprint, as `ssh-keygen -lf` prints it (`SHA256:...`). | | `storage.access_key_id` | The S3 access key. Instead of writing it here, you can set `SAFEGRD_S3_ACCESS_KEY` (or `AWS_ACCESS_KEY_ID`), or have SafeGrd hold the bucket credential. | | `storage.secret_access_key` | The S3 secret key. Instead of writing it here, you can set `SAFEGRD_S3_SECRET_KEY` (or `AWS_SECRET_ACCESS_KEY`). | | `storage.force_path_style` | Use path-style bucket addressing. MinIO needs it. | | `storage.layout` | `hosted`, with `type: s3`, reads SafeGrd's hosted storage bucket directly with that bucket's key. SafeGrd uses it for its own recovery. `list`, `restore` and `verify` work, and a backup is refused. To restore hosted backups without SafeGrd, keep a copy in your own bucket with `safegrd export` ([runbooks](https://safegrd.dev/docs/runbooks#without-safegrd)). | | `storage.iam_role_arn` | An IAM role the CLI assumes through STS to reach the bucket, in place of static keys. | ## copies Every backup can also be written to up to two more S3 buckets. Each entry is a full storage block with a name ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). If the project names buckets in the console, those replace this list. | Key | What it does | | --- | --- | | `copies[].name` | The copy's name: 1 to 32 lowercase letters, digits and dashes. `--from-bucket` and the console refer to the copy by this name. | | `copies[].type` | `s3`. It is the only kind of copy. | | `copies[].worm_mode` | Required: `COMPLIANCE`, `GOVERNANCE` or `NONE`. A copy does not inherit the first bucket's lock mode. | | `copies[].retention_days` | How many days each backup is locked in this bucket, counted from when it was taken. Defaults to the surface's. | | `copies[].expire_after_lock` | Once a day, the daemon deletes backups from this bucket whose lock has ended. It keeps the bucket's `min_keep` newest (default 3). Off unless you set it here. The first bucket's setting does not carry over. | | `copies[].bucket` | The bucket, with `region`, `endpoint`, `prefix`, `storage_class`, `force_path_style`, `access_key_id` and `secret_access_key` as in `storage`. The two keys take `env:VAR` or `file:/path`. A copy cannot use the same bucket and prefix as the first bucket. | ## encryption | Key | What it does | | --- | --- | | `encryption.public_key` | The Age recipient every snapshot is encrypted to (`age1…`). You can also set `SAFEGRD_PUBLIC_KEY`. | | `encryption.key_path` | The identity file that decrypts the snapshots, kept at mode 0600. The private key itself is never written into `config.yaml`. You can supply it with `SAFEGRD_PRIVATE_KEY` instead. | | `encryption.custody` | How the key is managed. `try`, `enroll` and `init` write it. `safegrd` is a SafeGrd-managed key: SafeGrd keeps it sealed and releases it only to your enrolled hosts. `local` is a customer-managed key: only you can decrypt these backups. With `safegrd`, enrolling again sends the key so that SafeGrd holds it for the new host. If this is empty, the key was put on the host by hand, and the CLI never sends it. | ## daemon Settings for the daemon ([Daemon](https://safegrd.dev/docs/daemon)). | Key | What it does | | --- | --- | | `daemon.interval` | How often the daemon looks for due work. Default `5m`. | | `daemon.state_dir` | Where the daemon keeps its state and locks. Default `~/.safegrd`. Do not put it on a network filesystem. | | `daemon.metrics_addr` | The address where the resident daemon serves Prometheus metrics at `/metrics`, such as `127.0.0.1:9847`. `daemon run --once` does not serve them ([the metrics](https://safegrd.dev/docs/daemon#metrics)). | | `daemon.ping_url` | A Healthchecks.io or Cronitor check URL. The daemon pings it after every pass over the surfaces, `--once` included. It pings the URL itself, or `URL/fail` when the pass could not run. Takes `env:VAR` or `file:/path`. | | `daemon.updates` | `console` (the default) or `off`. With `console`, the daemon downloads each new release and checks its signature. It installs the release when an owner or admin clicks Update in the console. From the console you can also send the host back to the release it ran before. Only hosts set up with `install.sh` update this way ([Updates from the console](https://safegrd.dev/docs/daemon#updates)). | | `daemon.export_to.bucket` | A bucket you own. When it is set, the daemon runs `safegrd export --to-bucket` on `daemon.export_to.schedule`, after that pass's backups. It copies every snapshot, still encrypted and with its lock date, and skips what is already there. The console shows on the host when the last copy was made. Set this or `daemon.export_to.dir`, not both ([Keep a copy of hosted backups](https://safegrd.dev/docs/runbooks#export-schedule)). | | `daemon.export_to.endpoint` | The bucket's S3 endpoint (`--to-endpoint`). Leave it out on AWS. | | `daemon.export_to.region` | The bucket's region (`--to-region`). Default `us-east-1`. | | `daemon.export_to.prefix` | The key prefix in the bucket (`--to-prefix`). Default `safegrd/snapshots`, which is the `prefix` the runbook's recovery config uses. | | `daemon.export_to.worm_mode` | The bucket's Object Lock mode (`--to-worm-mode`): `COMPLIANCE` (the default), `GOVERNANCE`, or `NONE` for a bucket without Object Lock. A bucket without Object Lock refuses a locked copy, and the export stops before it copies anything. | | `daemon.export_to.access_key_id` | An access key that can list and write to the bucket. Takes `env:VAR` or `file:/path`. Set it together with `secret_access_key`, or leave both out to use the standard AWS environment, as the command does. | | `daemon.export_to.secret_access_key` | Its secret. Takes `env:VAR` or `file:/path`. | | `daemon.export_to.force_path_style` | `true` for MinIO. It puts the bucket's name in the path instead of the host name. | | `daemon.export_to.dir` | A local directory to copy to, instead of a bucket (`--to-dir`). [Runbook 4](https://safegrd.dev/docs/runbooks#from-export) restores from it. | | `daemon.export_to.schedule` | How often the copy runs, written like a surface schedule: `@daily` (the default), `6h`, `0 3 * * *`. If a run fails, the daemon prints the error to stderr and tries again an hour later. A failed copy does not stop the backups. | | `daemon.updates_url` | A mirror the daemon downloads releases from, in place of `https://github.com/safegrd/cli/releases/download`. It has to serve `/v/`. Releases from a mirror get the same signature check. | ## defaults What every surface inherits unless it sets its own. | Key | What it does | | --- | --- | | `defaults.schedule` | `@hourly`, `@daily`, `@weekly`, or an interval such as `6h`. The shortest interval is one hour, so a schedule cannot fill the bucket with locked objects. | | `defaults.timezone` | Accepted, not acted on. Schedules run as intervals from the last success, not by wall-clock time. | | `defaults.keep_daily` | Lock the first backup of each UTC day for this many days ([GFS](https://safegrd.dev/docs/storage#gfs)). | | `defaults.keep_weekly` | Lock the first backup of each ISO week for this many weeks ([GFS](https://safegrd.dev/docs/storage#gfs)). | | `defaults.keep_monthly` | Lock the first backup of each month for this many months. | ## surfaces[] One entry for each surface the daemon backs up. Each entry appears on its own in the console and counts as one surface on your plan. | Key | What it does | | --- | --- | | `surfaces[].id` | A stable identifier. If you change it, SafeGrd sees a new surface. | | `surfaces[].type` | `postgres`, `mysql`, `mongodb`, `sqlite`, `files`, `email` or `laravel`. | | `surfaces[].name` | The name shown in the console. Defaults to the id. | | `surfaces[].schedule` | Overrides `defaults.schedule`. Takes an interval, a time of day (`at 02:00`) or a cron line ([schedules](https://safegrd.dev/docs/daemon#schedules)). | | `surfaces[].skip_if_unchanged` | With `true`, an incremental run that finds nothing changed since the last backup writes no new one. The console shows the surface as unchanged since that backup, instead of Overdue. Works with your own bucket and local storage only ([details](https://safegrd.dev/docs/daemon#unchanged)). | | `surfaces[].clock` | The clock that a time of day or cron line in `schedule`, and `blackout`, follow: `utc` (the default) or `local`. | | `surfaces[].blackout` | Times of day when no scheduled backup starts, such as `["22:00-02:00"]`. They are read in UTC unless you set `clock: local`. A backup that falls due waits for the window to end ([details](https://safegrd.dev/docs/daemon#blackout)). | | `surfaces[].retention_days` | Overrides `storage.retention_days` for this surface. | | `surfaces[].keep_daily` | Overrides `defaults.keep_daily`. | | `surfaces[].keep_weekly` | Overrides `defaults.keep_weekly`. | | `surfaces[].keep_monthly` | Overrides `defaults.keep_monthly`. | | `surfaces[].pre_backup` | A shell command the daemon runs before each backup of this surface, for example to flush a cache or pause a writer. It has 10 minutes. If it fails, no backup is taken, and the hook's own output in the daemon's log says why. The daemon sets `SAFEGRD_SURFACE_ID` and `SAFEGRD_SURFACE_TYPE` for it. | | `surfaces[].post_backup` | A shell command the daemon runs after every attempt, whether it succeeded or not, so you can resume whatever `pre_backup` paused. `SAFEGRD_BACKUP_STATUS` is `success` or `failed`, and `SAFEGRD_SNAPSHOT_ID` names the snapshot. If the command fails, the failure is reported and the backup stands. | | `surfaces[].pre_backup_on_error` | What happens when `pre_backup` fails. `skip` (the default) takes no backup. `retry` runs the hook again after 10 seconds, then after 20, and skips if it still fails. `continue` takes the backup anyway and logs a warning. | | `surfaces[].ping_url` | A Healthchecks.io or Cronitor check URL for this surface. The daemon pings `URL/start` before each backup, `URL` after it, and `URL/fail` after one that failed or was not recorded. The body is one line saying what happened. Takes `env:VAR` or `file:/path`. A failed ping is logged as a warning and changes nothing else. | | `surfaces[].post_drill` | A shell command the daemon runs after each Fire Drill of this surface. `SAFEGRD_DRILL_STATUS` is `passed`, `shallow`, `failed` or `blocked`, and `SAFEGRD_SNAPSHOT_ID` names the snapshot. `SAFEGRD_SANDBOX_URL` is set while the sandbox still holds the restore. If the command fails, the failure is logged and nothing else changes. | | `surfaces[].database_url` | A database surface's connection URL. Write it out, or use `env:VAR` or `file:/path` to keep it out of this file, which is how `init --database-url` writes it. If the host should not keep the password in a variable or a file, use `credential`. A SQLite surface gives its file here, as `sqlite:///var/lib/app/app.db`. There is no `path` key. | | `surfaces[].credential.from` | Where a database or mailbox surface's credential comes from. For a database it is the whole connection URL, and for a mailbox the password. With `safegrd`, SafeGrd holds it encrypted and gives it only to this host when the surface backs up. Nothing is set on the host, and it is never written to this file. With `env`, `command` or `file`, it stays on this host, as described below. A credential has one source. The CLI refuses a `credential` block together with a `database_url`. If the variable, command or file it names cannot be read, that surface fails, and the CLI does not fall back to anything else. | | `surfaces[].credential.name` | With `from: env`, the environment variable that holds it. Under a service, set it in the service's environment ([details](https://safegrd.dev/docs/daemon)). | | `surfaces[].credential.run` | With `from: command`, a command that prints the secret, such as a secret manager's CLI. It runs under `/bin/sh` for up to 30 seconds. If it fails, that surface fails ([details](https://safegrd.dev/docs/daemon)). | | `surfaces[].credential.path` | With `from: file`, a file whose contents are the secret, such as a mounted secret. | | `surfaces[].roots` | The directories a files surface backs up. A `laravel` surface takes one: the app's root, where `artisan` and `.env` are ([Laravel](https://safegrd.dev/docs/surfaces/laravel)). | | `surfaces[].excludes` | Glob patterns for what a files surface leaves out. On a `laravel` surface they are paths from the app's root, such as `storage/app/exports`. | | `surfaces[].migrations_table` | The table where a `laravel` surface's app records its migrations, if `config/database.php` names one other than `migrations`. The drill checks that table. | | `surfaces[].format` | How a files or database surface is stored: `repo` (incremental, each run uploads only what changed) or `tar` (one archive per backup). Unset means `repo`. A mailbox is always one archive. For a surface added in the console, the Backup type you chose there is written here. See [incremental file backups](https://safegrd.dev/docs/surfaces/files#incremental) and [databases](https://safegrd.dev/docs/surfaces/postgres#incremental). | | `surfaces[].erasures` | The tables, as `schema.table`, whose deleted rows are deleted again when you restore an older backup. It adds a trigger to each table and a `safegrd` schema to the database. For PostgreSQL, with local storage, your own bucket or hosted storage. See [the erasure record](https://safegrd.dev/docs/surfaces/postgres#erasures). | | `surfaces[].change_log` | Makes a PostgreSQL surface skip reading tables that nothing has written to since its last run. It adds a trigger to each table and a `safegrd` schema to the database. See [the change log](https://safegrd.dev/docs/surfaces/postgres#change-log). | | `surfaces[].roles_without_passwords` | Backs up the roles a PostgreSQL surface's schema names without their password hashes. By default the hashes are included. A surface SafeGrd backs up for you has no such switch. See [roles, owners and grants](https://safegrd.dev/docs/surfaces/postgres#roles). | | `surfaces[].recovery_doc_sealed` | Encrypts the recovery document written beside each snapshot to the surface's recipient. By default it is plain text, so you can read it without SafeGrd. It holds no secret. See [the recovery document](https://safegrd.dev/docs/storage#recovery-doc). | | `surfaces[].compression` | How hard the surface's backups are compressed, from `zstd:1` (fastest) to `zstd:19` (smallest). Every level writes the same format, so a restore needs no setting. Default: zstd's own default, about `zstd:3`. | | `surfaces[].parallel` | How many connections an incremental PostgreSQL or MySQL backup reads tables over at once, all in one snapshot. For an incremental MongoDB backup, how many collections it reads at once. A restore or a sandbox drill loads tables over as many connections, capped by the server's free connections (for MongoDB, insertion workers per collection). Such a restore is not one transaction. If it fails, drop the target and restore again. MySQL reads in parallel only with the `FLUSH_TABLES` or `RELOAD` privilege. Without it, the backup reads over one connection and says why. A full backup (one archive) reads and restores one table at a time. Default 4. For PostgreSQL, 1 loads the restore in one transaction. | | `surfaces[].one_filesystem` | With `format: repo`, stay on each root's filesystem. Default yes when a root is `/`, no otherwise. | | `surfaces[].watch.url` | With a `watch:` block, a PostgreSQL, MySQL or MongoDB surface checks the backups another tool writes, such as Coolify's, instead of backing up a database. Set this to `s3://bucket/prefix` or a directory on the host. Set the surface's `schedule` to how often that tool writes ([PostgreSQL](https://safegrd.dev/docs/surfaces/postgres#watch), [MySQL](https://safegrd.dev/docs/surfaces/mysql#watch), [MongoDB](https://safegrd.dev/docs/surfaces/mongodb#watch)). | | `surfaces[].watch.endpoint` | The bucket's S3 URL, for a store other than AWS S3 (Cloudflare R2, Backblaze B2, MinIO). | | `surfaces[].watch.region` | The bucket's region, as the store shows it. | | `surfaces[].watch.access_key` | An access key id that can list and read the prefix, as `env:VAR` or `file:/path`. It needs no write or delete permission. | | `surfaces[].watch.secret_key` | Its secret access key, as `env:VAR` or `file:/path`. | | `surfaces[].watch.match` | A glob that a backup's file name must match, such as `pg-dump-*.dmp`, for when the location holds other files too. | | `surfaces[].object_store.endpoint` | When a files surface's one root is `s3://bucket/prefix`, it reads that bucket over the S3 API ([details](https://safegrd.dev/docs/surfaces/files#bucket)). This is the store's S3 URL. For Supabase Storage it is `https://PROJECT.storage.supabase.co/storage/v1/s3`. Leave it empty for AWS S3. | | `surfaces[].object_store.region` | The bucket's region, as the store shows it. | | `surfaces[].object_store.access_key_id` | The store's access key id, as `env:VAR` or `file:/path`. | | `surfaces[].object_store.secret_access_key` | The store's secret access key, as `env:VAR` or `file:/path`. It opens every bucket in the store, so it is never written into the config itself. | | `surfaces[].host` | A mailbox's IMAP server. Default `imap.gmail.com`. | | `surfaces[].port` | Its TLS port. Default 993. | | `surfaces[].username` | The mailbox login. | | `surfaces[].folders` | The folders to back up. Default: all but spam and trash. | | `surfaces[].ca_file` | A PEM bundle to trust for the mail server's certificate, on top of the system roots. Use it for a server behind a private CA. Default: `SAFEGRD_EMAIL_CA_FILE`, if set. | | `surfaces[].storage` | A whole `storage` block for this surface alone, with the same keys as above. | | `surfaces[].copies` | Which of the `copies` this surface writes to, by name. Leave it out to write to all of them, or set `[]` for none. | | `surfaces[].encryption` | An `encryption` block for this surface alone, to encrypt it to a different key. | | `surfaces[].drill.sandbox_url` | An empty database of the same engine that this surface's Fire Drills restore into. The drill refuses one that holds tables, and empties it afterwards. Without it, on a plan with sandbox drills, a Postgres surface restores into a throwaway cluster that the daemon starts from the host's PostgreSQL server, if one is installed. Otherwise drills run in memory. | | `surfaces[].drill.source` | `primary` (the default) drills the surface's own bucket. `rotate` drills the next of its buckets each time, so every copy is restored in turn. | | `surfaces[].drill.checks[].name` | The name the drill record and the console show for one of your own checks. The check runs against the sandbox after each drill that restores into a database. If a check fails, the drill fails ([details](https://safegrd.dev/docs/verify#checks)). | | `surfaces[].drill.checks[].sql` | A query, run in a read-only transaction. The first column of its first row is compared with `expect`. | | `surfaces[].drill.checks[].expect` | The value the query must return, or a comparison: `"0"`, `"true"`, `"> 0"`, `">= 1000"`, `"!= 0"`. Numbers compare as numbers. | | `surfaces[].drill.checks[].command` | Use in place of `sql`: a command run through `sh -c`, with `SAFEGRD_SANDBOX_URL`, `SAFEGRD_SNAPSHOT_ID` and `SAFEGRD_SNAPSHOT_CREATED_AT` (when the backup was taken, RFC 3339 UTC) set. The check passes if it exits 0. | | `surfaces[].drill.checks[].timeout_seconds` | How long the check may run. Default 60 for a query, 300 for a command. | | `surfaces[].drill.keep_failed_sandbox` | Set it to keep the sandbox for 24 hours when a drill restores the snapshot and then fails a check (the digest or the row counts). The drill prints where the sandbox is, without the password. A `sandbox_url` database holds back the next drill until the 24 hours are up, and the first drill after that empties it. A throwaway cluster keeps running, and the daemon removes it after the 24 hours. A restore that fails is rolled back, so there is nothing to keep. The drill's record says the sandbox was kept. Default `false`. | | `surfaces[].wal.window_days` | How many days back a surface's point-in-time archive can restore to. Its base backups and bundles are locked for that long ([PostgreSQL](https://safegrd.dev/docs/surfaces/postgres#pitr), [MySQL](https://safegrd.dev/docs/surfaces/mysql#pitr), [MongoDB](https://safegrd.dev/docs/surfaces/mongodb#pitr)). The `wal:` block turns WAL archiving on, and `wal: {}` keeps every default. Default 7. | | `surfaces[].wal.base_every_days` | How often the daemon takes a base backup. For PostgreSQL that is `pg_basebackup`. For MySQL and MongoDB it is a `mysqldump` or `mongodump` that records its position in the log. A restore replays every WAL file since its base, so more frequent bases make a busy database restore faster. Default 7. | | `surfaces[].wal.incremental_every_days` | How often the daemon takes an incremental base backup between full ones. It holds only the blocks changed since the newest full, and is locked until that full is. Needs PostgreSQL 17 or newer with `summarize_wal = on`. Otherwise the daemon says why and waits for the next full base. Default 0, full bases only. | | `surfaces[].wal.bundle_mb` | Once a day, the daemon locks the unlocked WAL into one bundle. It does so sooner once the WAL passes this many MiB. Default 1024. | | `surfaces[].wal.drill_every_days` | How often the daemon drills the archive. A PostgreSQL archive goes into a throwaway cluster on the host, run by a PostgreSQL server of the same major version. A MySQL or MongoDB archive goes into the surface's `drill.sandbox_url`. The drill restores to the newest restore point, counting each table, and also to the end of the archive. A drill that fails or cannot run is tried again a day later. Default 7. | | `surfaces[].wal.interval_seconds` | For a SQLite surface, how often the resident daemon copies the database's newly committed WAL frames. A restore gets to within this many seconds of the moment you ask for ([details](https://safegrd.dev/docs/surfaces/sqlite#pitr)). Default 1. For a MySQL or MongoDB surface, how often the daemon pushes what the binary log or oplog committed since the last push. Default 5. | | `surfaces[].wal.server_id` | For a MySQL surface, the replica id the daemon gives the server when it reads the binary log. Set it if the id derived from the surface id is already a replica's own `server_id`. Default: derived from the surface id. | | `surfaces[].wal.checkpoint_mb` | For a SQLite surface, the WAL size at which the daemon checkpoints the database. While the daemon follows the WAL, the application's own checkpoints cannot shrink it past what the daemon has copied. Default 4. | | `surfaces[].wal.spool` | For PostgreSQL. When on, `archive_command` returns once a WAL file is copied into the spool and synced, and the daemon uploads from the spool ([details](https://safegrd.dev/docs/surfaces/postgres#pitr)). Default off. | | `surfaces[].wal.spool_dir` | Where the spool lives. The user PostgreSQL runs as writes there, and the daemon's user reads and deletes there. Default `wal-spool` in the state directory. | | `surfaces[].wal.parallel` | How many spooled WAL files the daemon uploads at once, and how many a restore fetches ahead. Default 4. | | `surfaces[].wal.queue_max` | The most WAL that can wait on the host for the archive, such as `8GB`. Past that, the oldest is dropped without being archived, so `pg_wal` cannot fill the disk. `wal status` lists what was dropped, and SafeGrd sends an alert. Default: no limit. | --- Source: https://safegrd.dev/docs/surfaces # Surfaces A **surface** is one thing you back up: a database, a file tree or a mailbox. Each Fire Drill tests one surface. Every surface goes through the same steps. The data is streamed, compressed with zstandard, encrypted to your age public key and written to locked storage. Only the first step (how the data is read) and the last (how a drill checks it) change from one kind of surface to the next. The stream is encrypted on the host that read it, so no plaintext crosses a third-party network. It is not written to disk either, with one exception. A SQLite surface is first copied to a temporary file on the same host, readable only by its owner ([SQLite](https://safegrd.dev/docs/surfaces/sqlite)). The file is deleted as soon as it is uploaded. ## The surfaces | Surface | What is captured | The host needs | | --- | --- | --- | | [PostgreSQL](https://safegrd.dev/docs/surfaces/postgres) | Schema from `pg_dump`, rows over binary `COPY`, one snapshot | `pg_dump` at least as new as the server | | [MySQL and MariaDB](https://safegrd.dev/docs/surfaces/mysql) | `mysqldump --single-transaction` | The MySQL client | | [MongoDB](https://safegrd.dev/docs/surfaces/mongodb) | `mongodump --archive` | The MongoDB Database Tools | | [SQLite](https://safegrd.dev/docs/surfaces/sqlite) | One committed state, through `VACUUM INTO` | Nothing | | [Files](https://safegrd.dev/docs/surfaces/files) | A directory tree, incremental: each run uploads only what changed | Read access to the tree | | [Email](https://safegrd.dev/docs/surfaces/email) | Every message of an IMAP mailbox, as RFC 5322 | The mailbox's password or app password | | [WordPress](https://safegrd.dev/docs/surfaces/wordpress) | The site's database and files in one snapshot, from a plugin. Each backup uploads what changed. | The SafeGrd Backup plugin. No host to set up. | | [Laravel](https://safegrd.dev/docs/surfaces/laravel) | The app's database, `.env` and `storage/app` in one snapshot. Each backup uploads what changed. | The CLI on the app's host, reading its `.env` | If you need something that is not in the table, such as Microsoft 365 mail, MSSQL, Redis, ClickHouse, Elasticsearch, Windows hosts or Kubernetes volumes, [request a surface](https://safegrd.dev/request-surface). ## What a Fire Drill proves, per surface Every Fire Drill downloads the snapshot from your bucket and decrypts it with your key. What it checks after that depends on the surface. | Surface | What the drill checks | | --- | --- | | PostgreSQL | The table count and every table's row count match the manifest, each table's binary COPY stream is readable, and the extensions are present. By default the drill runs in memory, with no database running anywhere. That checks the data but never runs the schema. To prove the schema loads, run a sandbox drill: with `--sandbox-target`, the snapshot is restored into a scratch database you name and counted there. The daemon does the same for a surface with `drill.sandbox_url`. On a plan with sandbox drills, a surface without one restores into a throwaway cluster that the daemon starts from the host's PostgreSQL server ([Daemon](https://safegrd.dev/docs/daemon)). | | MySQL and MariaDB | The dump ends with its completion line. Every table the manifest names is in it, with exactly the recorded number of rows. In memory, that checks the data. A sandbox drill (`--sandbox-target mysql://…` or `drill.sandbox_url`) loads the dump into an empty database, counts every table, and empties the sandbox again. | | MongoDB | Every collection and view in the archive reached its end. Each collection's documents match the checksum `mongodump` wrote and the count in the manifest. A sandbox drill restores into an empty database with `mongorestore`, counts every collection, and drops them again. | | SQLite | A real restore on every drill. The database file is written out and opened by the SQLite engine. It has to pass `PRAGMA integrity_check`, and every table's row count has to match the manifest. The file is deleted afterwards. | | Files | The whole snapshot is restored into a scratch directory, and every file has to match its SHA-256. The digest recomputed from the restored files has to match the one recorded at backup time. A `--format tar` archive is checked in memory against the manifest sealed inside it. | | Email | The message count matches the manifest, every message parses as RFC 5322 mail, and every message's SHA-256 matches. Nothing is written to disk. | | Laravel | The database passes its engine's drill and every file matches its SHA-256. The `.env` in the snapshot holds an `APP_KEY` that Laravel's cipher accepts, and the dumped `migrations` table agrees with the migration files. | A drill does not replace a [rehearsed restore](https://safegrd.dev/docs/runbooks), which shows that a person can get the data back. A drill shows that every snapshot reads back whole, without anyone having to remember to check. ## Running several surfaces on one host To back up more than one thing from a host, list the surfaces in the config file. The daemon runs each on its schedule. ``` surfaces: - id: "app-primary" type: "postgres" schedule: "@daily" credential: from: env name: "APP_DATABASE_URL" - id: "user-uploads" type: "files" schedule: "6h" roots: ["/var/www/uploads"] ``` Each entry registers as its own node and counts as one surface against your plan quota. Keep each `id` stable, because state, locks and snapshot history are keyed on it. A config with no `surfaces:` list still loads as one implicit Postgres surface, so an existing install needs no edit. The full shape is in [Daemon](https://safegrd.dev/docs/daemon). ## Adding a surface to a host that is already running The host's config is the one list of what it backs up, so you add a new surface there. You can do it from the console or by hand. In the console, click **Add surface** on the host's row under **Nodes**, and give the surface's name, its schedule and where its credential comes from. Nothing changes on the host until you run `safegrd claim` there: ``` safegrd claim ``` `safegrd claim` adds the surfaces named for that host to its config. It only adds. A surface the config already has is left as it is, and the previous file is kept as `config.yaml.bak`. Running it again adds nothing. If SafeGrd holds the credential, you type it once in the console and it is never written to the file. By hand, add an entry under `surfaces:`, as above. Either way, a running daemon reads the changed config at the start of its next pass and backs up the new surfaces then. To start one straight away, click _Back up now_ on the surface's row. With no daemon running, `safegrd daemon run --once` takes the first backups. --- Source: https://safegrd.dev/docs/surfaces/databases # Databases SafeGrd backs up four database engines, connecting with a read-only role you create. Each backup reads one consistent snapshot with the engine's own tools. The host compresses and encrypts it before it goes to locked storage. Each engine has its own page. - Your database one consistent snapshot, read-only role - The engine's own tools pg_dump and COPY, mysqldump, mongodump - The host compresses, then encrypts with age - Storage locked under Object Lock | Database | What is captured | The host needs | | --- | --- | --- | | [PostgreSQL](https://safegrd.dev/docs/surfaces/postgres) | Schema from `pg_dump`, rows over binary `COPY`, one snapshot | `pg_dump` at least as new as the server | | [MySQL and MariaDB](https://safegrd.dev/docs/surfaces/mysql) | `mysqldump --single-transaction` | The MySQL client | | [MongoDB](https://safegrd.dev/docs/surfaces/mongodb) | `mongodump --archive` | The MongoDB Database Tools | | [SQLite](https://safegrd.dev/docs/surfaces/sqlite) | One committed state, through SQLite's online backup | Nothing | ## A first backup, and the next one The first backup each month uploads every table. Later backups read every table again and upload only what changed. Both runs below are real: a host backing up a PostgreSQL database of 70,000 rows to hosted storage. ## Where the connection URL comes from PostgreSQL, MySQL and MongoDB connect with a URL. You choose where each database's URL is kept, and you can mix the two options on one host. SQLite has no secret. Name the file as `database_url: sqlite:///var/lib/app/app.db`, with no `credential` block. The URL stays on the host the default The surface's `credential` block tells the host where to read it. Use `from: env` with a variable's `name`, `from: command` with a command that prints it (such as `op read` or `vault kv get`), or `from: file` with a `path`. SafeGrd sees the variable's name or the command, and never the URL. SafeGrd holds the URL from: safegrd SafeGrd keeps the URL sealed. The host fetches it over HTTPS when the surface backs up and keeps it in memory for that run. Enter it in the setup wizard, or under _Credential_ in the surface's panel in the console. ``` surfaces: - id: "app-database" type: "postgres" credential: # the URL stays on the host from: env name: "APP_DATABASE_URL" - id: "billing-db" type: "mysql" credential: # SafeGrd holds the URL from: safegrd ``` Give each surface its credential in one place. If a surface has both `from: safegrd` and a `database_url` on the host, the backup stops with a configuration conflict error. Letting SafeGrd hold the database credential does not give it the encryption key. You make those two choices separately ([Security and key custody](https://safegrd.dev/docs/security)). ## Each table across backups Open a database surface's panel in the console and choose the _Tables_ tab. It has one row per table across the last 30 backups. Each row draws the table's row count as a line. A filled dot marks a backup that a drill restored to that row, and a square marks one where it could not. Each row also shows: - the table's size and newest row at the newest backup - a _Changed_ mark, with when its definition last changed - what the last drill found for the table You can sort by name, rows, size or change, and filter by name. Open a row to see the table's columns and what you need to restore that table alone: the sequences it owns and the tables its foreign keys point at. `safegrd restore --table` names those and does not load them. The _Overview_ tab shows the surface's trend over the last 30 days, under _Trend_: - how much the backup grows a day, raw and as stored. For a surface on hosted storage, it also shows the day your organization's quota fills at that rate. - the last run's time and speed against the median of the last 30 runs, and the runs that took twice the median or more - the tables behind what an incremental backup uploaded - what a run noticed compared with the backup before it: a mail folder the server rebuilt, directories gone, an extension gone from the top of the list The trend is computed from the figures each backup already reports, kept as one row per day. It is never read from the backup's contents. Each run is compared with the same backup that Threat Shield compared it with. The columns come from the backup. Each PostgreSQL, MySQL, MariaDB and SQLite backup records every table's columns and a hash of its definition, read inside the backup's own transaction. A MongoDB backup records each collection's top-level fields across all its documents, with their BSON types, as the collection's columns. Threat Shield compares them between backups and warns you when a table loses a column ([what each snapshot records](https://safegrd.dev/docs/threat-shield#record)). ## What a Fire Drill checks | Database | What the drill checks | | --- | --- | | [PostgreSQL](https://safegrd.dev/docs/surfaces/postgres) | The number of tables, every table's rows, the extensions and the [roles the schema names](https://safegrd.dev/docs/surfaces/postgres#roles), against what the backup recorded. In memory, it reads every row back with no database server. A sandbox drill (`--sandbox-target`, or `drill.sandbox_url` on the host) loads the schema and the rows into an empty database, so the constraints and indexes get built too. On a plan with sandbox drills, a host with no `drill.sandbox_url` restores into a throwaway cluster that it starts from its own PostgreSQL server ([Daemon](https://safegrd.dev/docs/daemon)). | | [MySQL and MariaDB](https://safegrd.dev/docs/surfaces/mysql) | The dump ran to its end, and every table the backup recorded is in it, with its structure and its row count. A sandbox drill (`--sandbox-target mysql://…`, or `drill.sandbox_url`) loads the dump into a scratch database, counts the tables there and drops it ([the sandbox account](https://safegrd.dev/docs/surfaces/mysql#sandbox)). | | [MongoDB](https://safegrd.dev/docs/surfaces/mongodb) | The archive ran to its end, and each collection's document count and checksum match what `mongodump` recorded. A sandbox drill restores the archive into a temporary database with `mongorestore`, counts each collection there and drops them ([the sandbox user](https://safegrd.dev/docs/surfaces/mongodb#sandbox)). | | [SQLite](https://safegrd.dev/docs/surfaces/sqlite) | A full restore. The database file is written out, opened and run through `PRAGMA integrity_check`. Every table's rows are counted against the backup, then the file is deleted. | The [table on the Surfaces page](https://safegrd.dev/docs/surfaces#drills) compares the drills for every kind of surface, files and mail included. --- Source: https://safegrd.dev/docs/surfaces/postgres # PostgreSQL SafeGrd backs up PostgreSQL 13 and later from a read-only role. By default each backup uploads only what changed since the last one. Supabase, Amazon RDS and Neon each have a section at the end of this page. ``` # PostgreSQL 13 and later; the test suite runs 16 safegrd backup --database-url "$DATABASE_URL" --retention-days 14 ``` The schema comes from `pg_dump`: types, tables, functions, views, indexes, constraints and triggers, exactly as Postgres describes them. The rows are copied over the wire with binary `COPY`, table by table, in chunks. A table larger than the host's memory backs up without staging anything on disk. Both are read from one snapshot, so if the database is written to during the backup, the rows, row counts and sequence positions still agree. - Install a `pg_dump` at least as new as the server. It comes with the PostgreSQL client package: `postgresql-client-16` on Debian and Ubuntu, `brew install libpq` on macOS. SafeGrd finds versioned installs under `/usr/lib/postgresql` and Homebrew's keg-only `libpq` by itself. To name one explicitly, set `SAFEGRD_PG_DUMP`. SafeGrd passes it the password in the environment, so the password is not on its command line. - Without `pg_dump`, the backup still copies the rows, but the schema is incomplete. The CLI warns you. Fire Drills fail, because constraints, views, triggers and enum types cannot be restored without it. - You do not need a superuser unless an extension does. The owner role a provider gives you (Amazon RDS, Cloud SQL) can back up, drill and restore. It creates trusted extensions such as `pgcrypto` and `citext` itself, and leaves the provider's own extensions and system tables alone. If an extension needs a superuser, install it in the target database before you restore. - Restore into an empty database. A Supabase project is never empty, so restore a Supabase backup into Supabase's PostgreSQL image ([how](https://safegrd.dev/guides/supabase-backup-github-actions#restore)). - When you restore the whole database, the restore also applies what was set on the database itself: its settings (`ALTER DATABASE ... SET search_path`), each role's settings in it (`ALTER ROLE ... IN DATABASE`), and who may connect. If the restoring role may not make a setting, such as another role's, the restore lists it for you to run as a superuser. - You can restore a backup into a newer major version, which is one way to upgrade. Back up the old server and restore into an empty database on the new one. SafeGrd's tests restore a PostgreSQL 14 backup into 18 with enum types, identity and generated columns, views, partial indexes and PL/pgSQL functions. Identity and serial columns carry on from the restored rows. Restoring into an older major version is not tested. A drill on a database surface checks table counts, row counts, column counts and installed extensions, and that every role the schema names is in the backup. See [Restore and Fire Drills](https://safegrd.dev/docs/verify). Large objects (`pg_largeobject`, made with `lo_import` or `lo_create`) are in the backup, read under the same snapshot as the tables. A restore brings each one back under its own oid, with its bytes and its owner, so the rows that point at them work again. A Fire Drill checks every object's bytes against the digest the backup recorded. With `--to-sql`, each one becomes a `lo_create` file that `load.sql` runs after the tables. Grants on a large object (`GRANT SELECT ON LARGE OBJECT`) are not restored, so grant them again after a restore. A database that uses pgvector backs up and drills like any other. SafeGrd's tests back up a table of `vector` columns and run a sandbox drill that restores every row. Install pgvector in the database you drill or restore into. ## Roles, owners and grants The schema is dumped with each object's owner, grants and default privileges, and each table's policies name the roles they apply to. SafeGrd backs up the roles these name (owners, grantees, policy roles) in `roles.sql` beside the schema, as `pg_dumpall --roles-only` writes them, with their password hashes. The cluster's other roles are not backed up. Before the schema runs, a restore creates the roles the target does not have and prints `Created role` for each one. A created role is never a superuser or a replication role. Its login and password are as they were, so an application role logs in on the new cluster as it did on the old one. `safegrd restore --no-owner` restores without owners and grants, so every object belongs to the restoring role. It still creates the roles, because a policy cannot be restored without them. If the restoring role is not a superuser, the restore skips the owners and grants it may not set: an owner it is not a member of, or a grant on an object it does not own. It says how many it skipped and gives those objects to the restoring role. The roles it creates get no `CREATEDB`, `CREATEROLE` or `BYPASSRLS`, because only a role that holds them may set them. To keep every owner and attribute, restore as a superuser. The daemon's sandbox drill creates the roles the same way and drops them with the sandbox. If the cluster does not have a role the schema names, the role the sandbox URL connects as needs `CREATEROLE`. `verify --sandbox-target` keeps what it restored, roles included, so you can look at it. To leave the password hashes out, set `roles_without_passwords: true` on a surface or pass `--roles-without-passwords` to `safegrd backup`. A restore then creates the roles without a password. The same happens when the backup role cannot read `pg_authid` (an RDS master user, Supabase's `postgres`), and the backup says so. A surface SafeGrd backs up for you has no such switch and keeps the hashes. The in-memory drill fails a snapshot whose schema names a role the snapshot does not carry, because no fresh cluster could load it. Snapshots taken before 5 October 2026 carry no roles. The next backup does. ## pg_cron jobs If the database has `pg_cron`, each backup records its jobs (name, schedule, command, user) in the snapshot, encrypted. The commands stay in the snapshot and never reach SafeGrd's server, because a command can hold a key. - A restore creates each job again turned off, so nothing runs against restored data until you turn it on. It prints each job with its new id, then the statement that turns it on: `SELECT cron.alter_job(ID, active := true);`. - `safegrd restore --keep-cron-active` creates each job as it was at backup time. Use it when the restore replaces a lost server. - Jobs are created only if pg_cron schedules from the database you restore into (`cron.database_name`). A job is not created over one of the same name that the target already has. A `--schema` restore creates none. Each job that is not created is printed as a warning. - A backup records the jobs its role can see. A role that is not a superuser sees only its own jobs. - A sandbox drill creates the jobs turned off and fails if pg_cron refuses one. The sandbox runs no job and sends no `pg_net` request. ## Check the backups another tool makes If Coolify, a cron `pg_dump` or another tool already backs up the database, SafeGrd can check those backups where that tool stores them, instead of taking its own: ``` surfaces: - id: app-db type: postgres schedule: daily watch: url: s3://my-backups/data/coolify/backups/databases/team-0/app-db endpoint: https://ACCOUNT.r2.cloudflarestorage.com access_key: env:R2_KEY_ID secret_key: env:R2_SECRET ``` Set `schedule` to how often the other tool writes. On that schedule the daemon lists the location and reads the newest backup once. It records the backup's SHA-256 and counts every table's rows from the dump, then reports that as a backup, dated when the other tool wrote it. If a check finds the same backup as last time, it reports nothing. So the surface goes Overdue when the other tool has written nothing for twice its schedule, and Threat Shield compares each backup's rows and size with the last one. A dump that does not open, or ends early, is a failed backup. A Fire Drill reads the backup again and checks that its SHA-256 is the one the check read. It loads the backup into the sandbox in one transaction with `pg_restore` or `psql`, and compares every table's rows. It leaves out the rows of pg_cron's job table, so nothing the backup scheduled runs. On the free plan the drill counts the backup again in memory. The certificate names the tool that made the backup. It proves the backup restores and holds the rows it held when SafeGrd first read it. SafeGrd never read the database, so the certificate cannot show that the dump holds all of it. - The key only needs to list and read the prefix. It stays on the host as `env:` or `file:`, and nothing the other tool wrote passes through SafeGrd's server. - `url` can also be a directory on the host. That is where Coolify keeps its backups when S3 is off. - SafeGrd reads `pg_dump` custom format (Coolify's `pg-dump-*.dmp`) and plain SQL, gzipped or not. A `pg_dumpall` dump is checked but only drilled in memory, because loading it creates roles and databases. The host needs `pg_restore` and `psql` beside `pg_dump`. - Each new backup is downloaded twice, once by the check and once by the drill. On AWS S3 you pay egress for that. Cloudflare R2 charges none. - `safegrd backup --surface app-db` runs a check now. `safegrd verify --surface app-db` drills the backup it read. - The same block on a `type: mysql` or `type: mongodb` surface checks that engine's dumps ([MySQL](https://safegrd.dev/docs/surfaces/mysql#watch), [MongoDB](https://safegrd.dev/docs/surfaces/mongodb#watch)). Under one Coolify tree, each surface reads only its own engine's files. ### Check them with no host In the console, click _Back up on SafeGrd_, pick the database's engine and tick _Check the backups another tool makes_. Then give the bucket, the prefix and a key that can list and read it. SafeGrd holds the key, encrypted, and uses it only on the machines it starts: a check on the surface's schedule, and the drills on your plan's cadence. A check that finds nothing new records nothing, so the surface still goes Overdue when the tool stops writing. The drill's machine gets the same key to read the backup again, and never a database credential. Every use of the key is in the key access log. ## The erasure record A backup taken on Monday holds the account a user deleted on Tuesday. Restore it on Wednesday and the account is back. Name the tables whose deleted rows are erasures, and a restore deletes them again before it commits: ``` surfaces: - id: app-db type: postgres database_url: env:DATABASE_URL erasures: [public.users, public.profiles] ``` ### How deletes are recorded The next backup adds a `safegrd` schema to the database, with a table `safegrd.erasures` and a trigger on each named table. When a row is deleted, the trigger records its primary key, the transaction and the time. It does not record the row. Each table needs a primary key. The backup's role must be able to create a schema in the database and triggers on the tables, which the owner of both can do. The daemon moves those records out of the database on every check-in, and every backup moves them first. They go to the surface's storage at `/erasures//`, encrypted to your key like the backups. Each one is locked for as long as a backup taken at that time could be kept. If a record cannot be moved, it stays in the table for the next try, and the console shows the storage as failing. ### What a restore does A restore reads the record before it writes anything. After it loads and counts the rows, it deletes every row erased after the backup was taken, along with whatever `ON DELETE CASCADE` removes with it. It prints how many rows it removed from each table. An erasure the backup already saw is not replayed, so a row inserted since with the same key is treated as a new row. If the record cannot be read (another bucket, an export, a missing key), nothing is restored. The error names `--without-erasures`, which restores anyway and says erased rows may be back. A sandbox drill replays the record after the counts are checked, and an in-memory drill opens it. If the record cannot be read, you find out from a drill before the day you restore. ### Limits and storage The record does not cover a `TRUNCATE`, tables you do not name, or copies of the data outside the database, such as files in Storage or logs. Do not name a table whose deletes are routine (sessions, carts), because every delete writes a record. The record works on local storage, your own bucket or hosted storage, in the default repo format. On hosted storage, SafeGrd signs each object's write and read as it does for a backup, and sets its lock to the longest any of your backups could be kept. SafeGrd reads none of it, because the record is encrypted to your key. `DROP SCHEMA safegrd CASCADE` removes the trigger and the table. An object of the record is deleted once its lock has ended and no backup of the surface is older than the newest erasure in it. `safegrd prune` deletes it in your own bucket and on local storage, and SafeGrd deletes it on hosted storage. If a bucket's policy refuses deletes, the bucket's lifecycle rule has to remove it. ## Each backup uploads what changed A PostgreSQL backup is a run of the surface's repository. Each table is one file of the run, cut into chunks of about 64 KiB. A later run in the same month uploads and locks only the chunks that changed. A table nobody wrote, or one that only grows at the end, adds almost nothing. A table updated at random rows all over can add its whole size on every run. The snapshot detail in the console shows what each table uploaded. Every run is a restore point: a restore brings the database back as that run read it. There is no restore point between two runs. For one archive per backup instead, pass `--format tar` or set `format: tar` on a surface. When you add a surface in the console, **Backup type** sets this: _Incremental_ or _Full_. On hosted storage, you can download a full backup as one file from the console. An incremental one restores on a host ([hosted storage](https://safegrd.dev/docs/storage#hosted)). [MySQL](https://safegrd.dev/docs/surfaces/mysql#incremental), [MongoDB](https://safegrd.dev/docs/surfaces/mongodb#incremental) and [SQLite](https://safegrd.dev/docs/surfaces/sqlite) databases are incremental by default too. ``` # the whole database, into an empty one safegrd restore --snapshot snap-20261004-091500-a1b2c3 --target "postgres://…/app_restored" # one table: into an empty table of the same definition, or created if it was dropped safegrd restore --snapshot snap-20261004-091500-a1b2c3 --table public.orders --target "postgres://…/app" # every version of one table, by run safegrd find --table public.orders ``` You can give `--table` more than once. The tables all come from one run, so they are consistent with each other. They load in one transaction, which commits only if every table matches its SHA-256 and its row count. The target's other tables stay as they are. If the target no longer has a table, for example because an agent dropped it, the restore creates it as the run defined it, with its sequences, constraints and indexes. Its sequences are set to where the run left them. A table the target still has must be empty and defined as it was. The command prints the run's time, the tables the restored ones point at that it did not load, and the sequences of an existing table that it did not move. `restore --table public.orders --version 3` restores the third version that `find --table` lists. To look at a table's backup beside the live one, use `--as`. `restore --table public.orders --as public.orders_restored` creates `orders_restored` with the live table's columns and no indexes, constraints or defaults, and loads the backup's rows into it. The live table stays as it is. It has to be defined as the backup recorded it. Compare the two with SQL, copy back what you need, and drop the copy. To put chosen rows straight back by primary key, use `--rows`. `--table` works on a full backup (`format: tar`) too, with the same checks. The restore reads the archive twice, once for its definitions and once for the chosen tables' rows, and checks each read against the backup's SHA-256. A created table's foreign keys are added after the rows, so you can restore tables that point at each other together. ## Skip tables nobody wrote A run reads every table, because a database has no file times to compare. On a large database that read takes minutes and loads the server on every run. With `--change-log`, or `change_log: true` on a surface, a run skips each table that nothing has written to since the last run: ``` safegrd backup --database-url "$DATABASE_URL" --change-log # [postgres-1a2b3c] 5 of 9 tables not read: the change log shows no write since the last run. ``` - It creates a `safegrd` schema in the database and a trigger on each table. After an insert, update, delete or truncate, the trigger logs the table and the transaction, once per transaction. The role taking the backup needs to own the tables (or hold `TRIGGER` on them) and be able to create a schema, so it cannot be the read-only role above. If the role cannot add the trigger to a table, that table is read on every run and the backup names it. - A table is skipped only when the log shows no write to it since the last run, its definition is unchanged, and a run read it within the last 7 days. The next run reads every table after a restore of the database to an earlier point, a failover to another timeline, a renamed enum label or a server upgrade. SafeGrd does not use statistics counters. - Each transaction adds one row to the log for each table it writes. That costs about 12 µs per table on a laptop running PostgreSQL 16. Rows are deleted once every backup that reads the log has seen them. - Needs PostgreSQL 13 or newer. Surfaces SafeGrd backs up itself do not offer it yet. - A restore leaves the log out. The copy has no `safegrd` schema and no trigger. - `DROP SCHEMA safegrd CASCADE;` removes the log and every trigger from the source. ## Restore to a point in time A backup restores the database as it was when the run started. If you also archive its write-ahead log (WAL), you can restore a PostgreSQL you run yourself to any second inside a window, 7 days by default. Supabase, Neon and RDS do not let you set `archive_command`, so this is for self-hosted PostgreSQL 13 or newer. Give the surface a `wal:` block: ``` surfaces: - id: appdb type: postgres database_url: env:DATABASE_URL wal: window_days: 7 ``` Then set these in `postgresql.conf` and restart PostgreSQL: ``` wal_level = replica archive_mode = on archive_command = 'safegrd --config /etc/safegrd/config.yaml wal push --surface appdb %p' ``` ### How the WAL is stored Each WAL file is compressed and encrypted on the host and written to your bucket unlocked. Once a day the daemon joins them into one locked bundle and deletes the unlocked copies, so up to a day of WAL is kept without a lock. Deleting them needs `s3:DeleteObjectVersion` on `wal/`. Without it, the copies stay beside the bundle and the daemon says so. On hosted storage the window is 7 days on Growth and 30 on Scale. SafeGrd sets every lock and deletes the WAL files a bundle holds, so the host does not hold a storage key. ### Base backups The daemon takes a base backup with `pg_basebackup` every 7 days (`base_every_days`). The surface's role needs `REPLICATION` (`ALTER ROLE safegrd_backup REPLICATION;`), and `pg_hba.conf` needs a `replication` line for the host. `safegrd wal base --surface appdb` takes one now. To spare the primary, point the surface's `database_url` at a standby. The base is taken there and its WAL comes from the primary's archive. The base records the standby and how far behind its primary it was. A base backup of a cluster with a tablespace outside its data directory is refused before anything is uploaded, and the error names each tablespace. The surface's scheduled backups still cover every tablespace. On a cluster with data checksums, a page that fails its checksum is read again before it is reported. A page that still fails does not fail the base. The host names the file and the page count, and `wal status` lists them under _Damaged pages_. ### Incremental base backups On PostgreSQL 17 and newer, the daemon can take an incremental base between the full ones. It holds only the blocks changed since the newest full, so a restore replays less WAL. Set `summarize_wal = on` in `postgresql.conf` (a reload is enough) and `incremental_every_days: 1` under `wal:`. Each incremental is taken from the newest full and locked until the same date as that full, so it never outlives the full it is rebuilt from. The daemon cannot take one on PostgreSQL 16 or older, with `summarize_wal` off, when the full is past its 7 days, when the window was raised since the full, or when this host did not take the full. Its log then says why, and the next base is a full one when it is due. `wal base --full` and `wal base --incremental` ask for one kind. A restore from an incremental unpacks it and its full beside the target directory, then combines them with `pg_combinebackup` of the same major version. It needs room for both and for the result. ### Run `archive_command` as the PostgreSQL user `archive_command` runs as the user PostgreSQL runs as, so the config and the key file have to belong to that user, at mode 0600. With the bucket and its key in the config, a push never waits on SafeGrd. If the host uses a bucket or a key SafeGrd holds, it asks SafeGrd for it on each push, and the push waits while SafeGrd does not answer. ### When a push fails While a push fails, PostgreSQL keeps the WAL in `pg_wal`, which grows until the push works again. The daemon checks `pg_stat_archiver` on every pass and logs _archive_command is failing_, and SafeGrd alerts your team. To keep a full disk from stopping the database, set `queue_max: "8GB"` under `wal:`. Once more WAL than that is waiting, the oldest is dropped without being archived. `wal status` lists it, and SafeGrd sends a `wal_dropped` alert. No restore can replay past it, so take a base backup once the archive is reachable. ### Spool WAL on a busy cluster A busy cluster can produce WAL faster than one upload per file can keep up. With `spool: true` under `wal:`, `archive_command` returns once the file is copied to the spool (`spool_dir`, default `wal-spool` in the state directory) and synced. The daemon uploads from the spool, `parallel` files at a time (4 by default). The user PostgreSQL runs as writes the spool, so the daemon's user needs to read and delete there. `safegrd wal drain` uploads it now. A restore fetches the same number of WAL files ahead. ### Restore points and WAL drills Before each daily bundle, the daemon marks a restore point and records each table's row count on both sides of it. To mark one now, before a migration for instance, run `safegrd wal point --surface appdb`. The role needs one more grant for it: ``` GRANT EXECUTE ON FUNCTION pg_create_restore_point(text) TO safegrd_backup; ``` Every 7 days (`drill_every_days`) the daemon tests the archive with the host's PostgreSQL server of the same major version. It restores into a throwaway cluster up to the newest restore point and counts each table against what the point recorded. Then it restores again to the end of the archive. `safegrd wal drill --surface appdb` runs it now. The console and the surface's certificate show the window and the last drill. ### Restore to a moment `safegrd wal status --surface appdb` shows the window a restore can reach, how much of it is locked, and any WAL file that is missing. Each base backup's record and each bundle's index is written twice, each copy with its checksum. If one copy is damaged, a restore reads the other. `--check` reads both copies of each and names any pair that differs. To restore, run this as the PostgreSQL user, with the same major version the base backup came from: ``` safegrd wal restore --surface appdb --to "2026-10-17 14:52:00" --pgdata /var/lib/postgresql/restore pg_ctl -D /var/lib/postgresql/restore start ``` The surface's **Point in time** tab in the console writes this command for a moment you pick. PostgreSQL replays the WAL to that time (UTC) and promotes. To stop somewhere other than a time: - `--to-point ` replays to a restore point. - `--to-lsn 16/B374D848` replays to a position in the WAL. - `--to-xid 1234567` replays to just after a transaction commits. - `--to latest` replays every archived WAL file. Use it to restore after you lose a host. PostgreSQL refuses a time at or after the last commit in the archive. `--exclusive` stops just before the target. `--timeline` follows a timeline other than the newest. A numbered timeline is checked against the archive's history of it before anything is written. If a restore would stop at a missing WAL file, it is refused before it unpacks anything, and the error names the file. Archiving is off on the restored cluster, so it does not write into the archive it came from. Turn it back on once this is the database you keep. On hosted storage, a host that replaces a lost one restores from the old host's archive with `--node `. To restore over the cluster's own data directory instead of an empty one, stop PostgreSQL and add `--delta`. The restore compares every file the base holds with the one on the target, byte for byte, and writes only those that differ. After a small fault, it does not write the whole cluster again. It refuses a directory PostgreSQL is running on, or one that belongs to another cluster. Files the base does not have are removed if they are inside PostgreSQL's own directories (`base`, `global`, `pg_wal` and the other `pg_*` directories), because PostgreSQL would read them as data. Anything else, such as a `log` directory, is kept and listed. If a delta restore stops partway, run it again. PostgreSQL will not start on the directory until it finishes. Keep the `backup_label` file the restore writes. The restore clears the checkpoint in `pg_control`, so PostgreSQL 13 to 17 start from the checkpoint the label names. They refuse to start if the label is deleted. ## Give SafeGrd a read-only role SafeGrd only reads your database. Give it a role of its own with no write permissions, so a leaked backup credential cannot change or delete your data. ``` # PostgreSQL 14 and later CREATE ROLE safegrd_backup LOGIN PASSWORD 'a strong password'; GRANT pg_read_all_data TO safegrd_backup; ``` `pg_read_all_data` is built in from PostgreSQL 14. It grants `SELECT` on every table, view and sequence and `USAGE` on every schema, including ones created later, so you do not have to grant each new table. PostgreSQL 13 has no such role. Grant access explicitly and set the default for tables created later: ``` # PostgreSQL 13: repeat the last three lines for every schema CREATE ROLE safegrd_backup LOGIN PASSWORD 'a strong password'; GRANT CONNECT ON DATABASE app TO safegrd_backup; GRANT USAGE ON SCHEMA public TO safegrd_backup; GRANT SELECT ON ALL TABLES IN SCHEMA public TO safegrd_backup; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO safegrd_backup; ``` Run the `ALTER DEFAULT PRIVILEGES` statement too. Without it, the role cannot read tables created later, and they are left out of future backups. `pg_dump` runs as the same role. Before you go on, check that the role cannot write: ``` psql "postgres://safegrd_backup:…@db.internal/app" -c "CREATE TABLE sg_probe(i int);" # expect: ERROR: permission denied for schema public ``` `safegrd doctor` checks this for every PostgreSQL surface the host can open, and `safegrd claim` checks the surfaces it adds. For a database SafeGrd backs up itself, the console checks it when you save the connection string and shows the same advice. The line _Surface role_ warns when the role is a superuser, can create roles or databases, or can write to any table. It names up to three of those tables and prints the two lines above as the fix. A second line, _Surface row security_, names the tables whose rows row-level security hides from the role. A role that is not a superuser cannot read password hashes, so roles are backed up without their passwords ([roles](https://safegrd.dev/docs/surfaces/postgres#roles)). A table the role cannot read does not fail the backup. The snapshot leaves it out, with its partitions and sequences, and the backup prints a warning that names the table and the grant that would include it. Views and row-level security policies that read a left-out table are left out too, and named in a second warning. A restore or `restore --to-sql` leaves out the foreign keys that point into those tables and says which ones. A Fire Drill of the snapshot passes. The snapshot records the list in `left-out.json`. Needs CLI `v0.0.39`. ### Row-level security To back up every row of a table with row-level security (RLS) policies, grant the role `BYPASSRLS` or back up as the tables' owner. `pg_read_all_data` does not bypass RLS by itself, so otherwise the backup holds only the rows the policies let the role see. SafeGrd checks each table before it copies it. The backup prints every table RLS applied to, with the rows copied and the table's estimated size. The console marks the table _Row-level security_ in the snapshot. ### Large objects `pg_read_all_data` does not reach large objects the role does not own. If a backup holds a large object it may not read, it says how many it left out, and its Fire Drill fails. Back up as their owner, or run `GRANT SELECT ON LARGE OBJECT TO safegrd_backup` for each one. ### Read replicas Point SafeGrd at a read replica to keep the backup's reads off your primary. It needs only read access. A backup of a standby holds what the standby had replayed, and the backup says how far behind it was: `Read from: a standby, its newest transaction committed 4s before the backup on the primary`. The snapshot's panel in the console shows the same. On a primary with no writes, that figure grows even though nothing is behind. Replay can cancel a long backup on a standby. If it does, raise `max_standby_streaming_delay` on the standby. ## A provider's own CA If a provider signs its servers with a CA of its own (RDS, Aiven, DigitalOcean, Supabase), you need that CA to verify them. Without it, `sslmode=require` encrypts the connection but checks no certificate. - On a host, add `?sslmode=verify-full&sslrootcert=/path/ca.pem` to the URL, or set `ca_file: /path/ca.pem` on the surface in the config, which sets both. - For a database SafeGrd backs up for you, paste the CA certificate when you add it, or later with **CA certificate** on the surface. SafeGrd opens the database with it before saving, and refuses a CA that does not verify the server. - A URL that also says `sslmode=disable`, `allow` or `prefer`, or names another `sslrootcert`, is refused. ## Supabase Supabase runs PostgreSQL with pre-installed schemas for auth, storage and realtime. To back it up from GitHub Actions with no server of your own, follow [Back up Supabase with GitHub Actions](https://safegrd.dev/guides/supabase-backup-github-actions). Or let SafeGrd take the backup: click _Back up on SafeGrd_ under Surfaces in the console ([how it runs](https://safegrd.dev/docs/verify#hosted)). Use the session pooler address below either way. In that form, _Sign in to Supabase_ signs you in at Supabase and lists your projects. Then choose one of two ways for SafeGrd to connect. ### Sign in through Supabase for each backup You type no password. SafeGrd holds the Supabase sign-in, encrypted. For each backup it asks Supabase for a read-only role whose password expires within minutes. The machine that takes the backup gets that role, and never the sign-in. Where the project allows it, the backup reads as `postgres` through that role. A table it still cannot read is [left out and named](https://safegrd.dev/docs/surfaces/postgres#left-out). The SafeGrd app asks Supabase for _Projects_ read and _Database_ read and write, because Supabase makes the role through a write call, even a read-only role. SafeGrd opens the database with the role before it adds the surface and before each backup. If row-level security hides rows from the role, SafeGrd refuses the sign-in, or fails that backup, and names the tables. If the database's firewall lets the backup machines in but not SafeGrd's server, that check cannot run, so allow both. ### Type the database password SafeGrd holds the connection string, as for any other database. The project's `postgres` user reads every row. ### Switch or reconnect You can switch a surface between the two from its Details. _Credential_ replaces the sign-in with a connection string, and _Sign in through Supabase_ does the reverse. If Supabase stops accepting the sign-in (you disconnected SafeGrd there, or removed the account's access to the project), the next backup fails before any machine starts and says so. Click _Reconnect Supabase_ on the surface to sign in again. ### Connection string Connect to the direct host, `db.[PROJECT-REF].supabase.co:5432`. It has only an IPv6 address unless you buy Supabase's IPv4 add-on. A host with no IPv6, such as GitHub's runners, uses the session pooler instead: `aws-0-[REGION].pooler.supabase.com:5432` with user `postgres.[PROJECT-REF]`. Do not use the transaction pooler on port 6543. It shares server connections between clients, which breaks binary `COPY` and the single snapshot the backup reads every table from. A backup through it, or through PgBouncer with `pool_mode = transaction`, is refused with an error that names the transaction pooler. Add `?sslmode=require` to the database URL. To verify the server too, download the CA certificate from the project's Database settings and set it as [the surface's CA](https://safegrd.dev/docs/surfaces/postgres#ca). ### Row-level security `pg_read_all_data` does not bypass RLS. Give the backup role `BYPASSRLS`, or grant explicit SELECT policies on tenant tables. Otherwise those tables back up with 0 rows. A project's own `postgres` user already has `BYPASSRLS`. ### Restore A Supabase project already holds Supabase's own schemas, and its `postgres` user is not a superuser, so you cannot restore into a project, even a new one. Restore into Supabase's PostgreSQL image as `supabase_admin`, following the guide's [restore section](https://safegrd.dev/guides/supabase-backup-github-actions#restore). Stock `postgres` images do not have extensions such as `supabase_vault`. ## Amazon RDS and Aurora (PostgreSQL) RDS restricts superuser access, and SafeGrd does not need it. Create a read-only role with `GRANT pg_read_all_data TO safegrd_backup;`. To keep dump traffic off your primary writer, point `database_url` at your read replica endpoint. SafeGrd only reads, so a replica is enough. Download Amazon's root certificate bundle (`global-bundle.pem`) and add `?sslmode=verify-full&sslrootcert=/path/global-bundle.pem` to the URL. For a database SafeGrd backs up for you, paste the bundle as [the surface's CA](https://safegrd.dev/docs/surfaces/postgres#ca). ## Neon Neon is serverless Postgres with compute autosuspend. Use the direct connection string without the `-pooler` suffix (for example `ep-xyz.region.aws.neon.tech:5432`), so binary streaming works. Neon requires TLS, so add `?sslmode=require` to the URL. If compute autosuspend is on, the first connection wakes the instance. Allow a few seconds for the cold start. --- Source: https://safegrd.dev/docs/surfaces/mysql # MySQL and MariaDB You can back up MySQL 8.0 and 8.4 and MariaDB 10.6 and 11.4 with one command. SafeGrd runs `mysqldump` on your host: ``` # MySQL 8.0 and 8.4, MariaDB 10.6 and 11.4 (each run by the test suite) safegrd backup --database-url "mysql://backup:$DB_PASSWORD@db.internal:3306/shop" --retention-days 14 ``` SafeGrd runs `mysqldump` (or `mariadb-dump`) with `--single-transaction`. It reads InnoDB tables from one consistent snapshot without locking them. The backup includes stored routines, triggers and events, and writes binary columns as hex. The dump streams into the encrypted backup as it is produced. SafeGrd counts rows from the dump stream itself, so the counts match the rows in the backup even while your app writes to the database. ## The account to back up as Give SafeGrd an account that reads and changes nothing. On MySQL 8.0.20 and newer: ``` CREATE USER 'safegrd_backup'@'%' IDENTIFIED BY '...'; GRANT SELECT, SHOW VIEW, TRIGGER, LOCK TABLES, EVENT ON shop.* TO 'safegrd_backup'@'%'; GRANT SHOW_ROUTINE ON *.* TO 'safegrd_backup'@'%'; ``` `TRIGGER` and `EVENT` let `mysqldump` read the database's triggers and scheduled events. `SHOW_ROUTINE` lets it read every stored procedure and function, including those other accounts created. If one of these grants is missing, the backup still runs and tells you what it left out. MariaDB and MySQL before 8.0.20 have no `SHOW_ROUTINE`. Grant the first line's privileges there, and the backup holds the routines the account can read. `safegrd doctor` warns you when a surface's account can write, and prints these grants. ### The sandbox for Fire Drills A sandbox drill loads the dump into a database used for nothing else, counts its tables and rows, then drops everything it loaded: tables, views, routines and events. Name that database with `drill.sandbox_url` on the host, or with `--sandbox-target`. It has to be empty before each drill. Give the account every right on it and none elsewhere: ``` CREATE DATABASE shop_sandbox; CREATE USER 'safegrd_sandbox'@'%' IDENTIFIED BY '...'; GRANT ALL ON shop_sandbox.* TO 'safegrd_sandbox'@'%'; ``` As the dump loads, SafeGrd rewrites every `DEFINER` in it to the sandbox account. A view, trigger or routine owned by another account then restores without `SUPER`. On a self-hosted MySQL with the binary log on (the default since 8.0), an account without `SUPER` can create a trigger or function only if the server sets `log_bin_trust_function_creators = 1`. Managed servers set it. Without it, the drill fails at the first trigger and tells you why. ## Incremental backups Each backup uploads only what changed since the last run. It is a run of the surface's repository, like a [PostgreSQL backup](https://safegrd.dev/docs/surfaces/postgres#incremental). SafeGrd cuts the dump at each table into a header, one file per table (its structure, rows and triggers), and a tail with the routines, events and view definitions. A table nobody wrote to since the last run gives the same bytes, so it uploads nothing. A table that changed uploads the 64 KiB chunks around the rows that moved. Every run still reads the whole database and is a restore point. The snapshot detail in the console shows what each table uploaded. An incremental backup reads its tables over four connections at once (`parallel` on the surface), all at the same moment of the database. To open them at one moment, the backup holds writes while it opens the connections and reads the schema. That usually takes under a second, and the backup prints how long. On MySQL 8.0.23 and newer it needs one more grant: ``` GRANT FLUSH_TABLES ON *.* TO 'safegrd_backup'@'%'; ``` On MariaDB and older MySQL, grant `RELOAD ON *.*` instead. If the grant is missing, or statements already running hold up the lock for 5 seconds, the backup reads one table at a time and tells you. `parallel: 1` always reads one table at a time, and so does a full backup (`--format tar`). On a 270 MB database, a run with no writes uploaded 0.1 MB, and 2,000 rows appended to one table uploaded 0.5 MB. 400 rows updated across a table of 300,000 uploaded 20 MB, the chunks around each row. A row inserted into or deleted from the middle of a table costs about the same as an update. `mysqldump` packs rows into `INSERT` statements by size, so every statement after that row shifts. For one archive per backup instead, use `--format tar`, or `format: tar` on a surface. The archive is the dump as `mysqldump` wrote it, and once you decrypt it, `mysql` loads it without SafeGrd. When you add a surface in the console, **Backup type** sets this. The change log (`--change-log`) works for PostgreSQL only. A MySQL run reads every table each time. ### Restore A run restores whole into an empty database: `safegrd restore --snapshot --target mysql://…`. SafeGrd checks every table's restored row count against the snapshot manifest. MySQL does not run DDL statements inside a transaction, so a failed restore can leave part of the schema behind. Drop and recreate the database before you try again. You can also restore one table into a database that has other tables: `safegrd restore --snapshot --table orders --target mysql://…`. This loads the dump's section for that table, with its rows and triggers. If the target has no such table, SafeGrd creates it as the backup defined it. If the table exists, it has to be empty and have the same columns. In the console, the Tables tab has **Restore table** for the same thing. A full backup (`tar`) is one dump, so it only restores whole. - Views, triggers and stored routines keep their original definer accounts in the dump. Recreating one with an arbitrary definer needs `SET_USER_ID`, which managed platforms such as RDS do not grant. So during a restore, SafeGrd rewrites each definer to the restoring account and reports how many definitions it changed. - Rows up to the source server's `max_allowed_packet` are backed up whole. Set `max_allowed_packet` on the restore target at least as high as on the source. - A trigger created through a multi-statement API can end in a semicolon that `mysqldump` exports as invalid SQL. SafeGrd finds these triggers during the backup, and a sandbox drill flags them. An in-memory drill does not parse SQL, so it cannot. ### Client tools, password and TLS The host needs the MySQL client (`mysql-client`, or `brew install mysql-client` on macOS). SafeGrd prefers the dump tool from the same project as the server. MySQL's client has also been tested against MariaDB 11.4. To name the tools yourself, set `SAFEGRD_MYSQLDUMP` and `SAFEGRD_MYSQL`. SafeGrd passes the password to the client in an owner-only options file, which it deletes when the run ends. The password is never on the client's command line. Choose TLS in the URL: - `?tls=true` encrypts and checks the server's certificate against the system's certificate authorities. - MySQL's own certificate is self-signed and fails that check. For a server that still uses it, add `?tls=skip-verify` to encrypt without checking, or `?ssl-ca=/path/ca.pem` to check it against your own CA. - `?tls=preferred` uses TLS when the server offers it. If your provider has its own CA (Aiven, DigitalOcean, RDS), name it on a host with `?ssl-ca=/path/ca.pem`, or with `ca_file` on the surface in the config. For a database SafeGrd backs up for you, paste the CA certificate when you add it, or later with **CA certificate** on the surface. Each backup then checks the server's certificate and host name against it. A URL that also says `tls=skip-verify` or `ssl-mode=PREFERRED` is refused, because it would check nothing. You can use `ssl-mode` in place of `tls`, as `mysql` takes it and as Aiven's URIs carry it. A URL that sets both to different values is refused. - `REQUIRED` is `tls=skip-verify`. - `VERIFY_CA` and `VERIFY_IDENTITY` are `tls=true`. - `PREFERRED` is `tls=preferred`. - `DISABLED` is `tls=false`. ## Check the backups another tool makes If Coolify, a cron `mysqldump` or another tool already backs up the database, SafeGrd can check those backups where that tool stores them instead of taking its own. Set it up as for [a PostgreSQL surface](https://safegrd.dev/docs/surfaces/postgres#watch), with `type: mysql`: ``` surfaces: - id: shop-db type: mysql schedule: daily watch: url: s3://my-backups/data/coolify/backups/databases/team-0/shop-db endpoint: https://ACCOUNT.r2.cloudflarestorage.com access_key: env:R2_KEY_ID secret_key: env:R2_SECRET ``` SafeGrd reads the plain SQL that `mysqldump` and `mariadb-dump` write, gzipped or not. That includes Coolify's `mysql-dump-*.dmp` and `mariadb-dump-*.dmp`. The check counts each table's rows from its `INSERT` statements. A dump without its `-- Dump completed` line counts as cut short. A dump of several databases (`--all-databases`, Coolify's "all databases" backup) is checked, but drilled in memory only, because loading it would write beside the sandbox. A Fire Drill loads the dump into the sandbox through the `mysql` or `mariadb` client, with each `DEFINER` set to the sandbox account. It then compares every table's rows. The client stops at the first error, and MySQL commits each statement as it runs. So a failed load leaves part of the dump in the sandbox, and the drill empties it afterwards. `drill.sandbox_url` must name a MySQL or MariaDB database. A sandbox of another engine is refused before anything connects. Under one Coolify tree, a `type: mysql` surface reads only the MySQL and MariaDB files. The PostgreSQL and MongoDB backups beside them need surfaces of their own type. If you have no host, add the surface in the console under _Back up on SafeGrd_ and tick _Check the backups another tool makes_, as [the PostgreSQL page](https://safegrd.dev/docs/surfaces/postgres#watch) describes. SafeGrd holds the bucket key, and checks and drills the backups on the machines it starts. ## Restore to a point in time A backup restores the database as it was when the run started. Add a `wal:` block to the surface and the resident daemon also follows the server's binary log. You can then restore the database to any second in a window, 7 days by default: ``` surfaces: - id: shop type: mysql database_url: mysql://safegrd_backup:...@db.internal:3306/shop wal: {} ``` The server has to keep a binary log in row format. MySQL 8.0 and newer do by default. On MariaDB, set this in `my.cnf` and restart: ``` [mysqld] log_bin = mysql-bin server_id = 1 binlog_format = ROW ``` Give the backup account above these grants so it can read the binary log. On MySQL: ``` GRANT REPLICATION SLAVE, REPLICATION CLIENT, FLUSH_TABLES ON *.* TO 'safegrd_backup'@'%'; ``` On MariaDB: ``` GRANT REPLICATION SLAVE, BINLOG MONITOR ON *.* TO 'safegrd_backup'@'%'; ``` `REPLICATION SLAVE` reads the log, and `REPLICATION CLIENT` (MariaDB: `BINLOG MONITOR`) reads where it ends. `FLUSH_TABLES` (`RELOAD` before MySQL 8.0.23) lets a base backup hold writes for the moment its snapshot opens, so the log position it records is exact. MariaDB reads that position from the snapshot and needs no lock. The binary log belongs to the whole server, so with these grants the account can read the changes to every database on it. The daemon reads the log as a replica does. Every 5 seconds (`interval_seconds`) it pushes the transactions committed since the last push to your bucket, compressed and encrypted on the host and unlocked. Once a day it joins them into one locked bundle and deletes the unlocked copies. On hosted storage, this comes with Growth (7 days) and Scale (30 days). The archive holds the whole server's log, encrypted, with every database's changes. A restore replays only this surface's database. A base backup is a `mysqldump` of the database that records the log position of its snapshot. The daemon takes one when it starts following the log, and again every 7 days (`base_every_days`). To take one now, run `safegrd wal base --surface shop`. ### Keep the log reachable - Only the resident daemon follows the log. `safegrd daemon run --once` from cron cannot, and tells you so. - When the daemon starts again, it carries on where the archive ends. If the server purged that part of the log in the meantime, the daemon tells you, starts from the server's current position and takes a new base. The window starts again from there. - Keep the binary log for at least a day (`binlog_expire_logs_seconds`, or `expire_logs_days` on MariaDB). Then a daemon that is down for a while leaves no gap. - `binlog_transaction_compression = ON` is refused, because the daemon cannot see which database a compressed transaction changes. - `safegrd doctor` checks that the log is on and in row format, the grants, how long the server keeps the log, and that this host has the `mysqlbinlog` a restore needs. ### How the daemon checks it Once a day the daemon records a restore point: the log position and every table's row count, read from the same snapshot. Every 7 days it restores the archive into the surface's `drill.sandbox_url`, to the newest restore point and to the end, and checks the counts. To run this check now, use `safegrd wal drill --surface shop`. Without a sandbox the check does not run, and the console tells you why. ### Restore to a moment Restore into an empty database: ``` safegrd wal restore --surface shop --to "2026-10-17 14:52:00" --target-url "mysql://restorer:...@db.internal:3306/shop_restored" ``` In the console, the surface's **Point in time** tab writes this command for a moment you pick. The command loads the newest base backup from before that moment (UTC). It then replays every transaction of this database committed by then, using the server's own `mysqlbinlog` and `mysql`. MySQL stamps each commit to the microsecond. MariaDB stamps it to the second, so a transaction committed in the same second as the moment you give is not replayed. - `--to-point ` restores to a restore point. - `--to latest` restores everything archived. - If the log has a gap before the moment you asked for, the restore is refused and names the gap. - `safegrd wal status --surface shop` shows the window. The account you restore with needs `ALL` on the target database and, on MySQL: ``` GRANT SESSION_VARIABLES_ADMIN, BINLOG_ADMIN ON *.* TO 'restorer'@'%'; ``` On MariaDB: ``` GRANT BINLOG REPLAY ON *.* TO 'restorer'@'%'; ``` The replay keeps a statement if it ran with this database as its default (`USE shop`), as MySQL replication's own database filter does. A statement that changed `shop` while another database was the default is not replayed. Row changes are kept by the table they change, whatever the default database was. PlanetScale does not give access to the binary log ("PlanetScale does not currently support binlog replication to external databases"), so point-in-time recovery is not available there. Regular backups of a PlanetScale database work as on any MySQL server. Amazon RDS for MySQL keeps no binary log for an outside reader until you set a retention, up to 168 hours: `CALL mysql.rds_set_configuration('binlog retention hours', 24);` ## Amazon RDS and Aurora (MySQL) RDS does not grant `SUPER`, and SafeGrd does not need it. - SafeGrd runs `mysqldump` with `--single-transaction` and `--set-gtid-purged=OFF`, which work with a standard user's privileges. - During a restore, SafeGrd rewrites object definers (for example `%` or `admin@%`) to the restoring account, so RDS does not reject them. - Back up from an Aurora or RDS read replica to keep table scans off your primary. --- Source: https://safegrd.dev/docs/surfaces/mongodb # MongoDB You back up MongoDB 7 and 8 with one command, through `mongodump`, and restore a database with another. ``` # MongoDB 7 and 8 (each run by the test suite) safegrd backup --database-url "mongodb://backup:$DB_PASSWORD@db.internal:27017/app?authSource=admin" --retention-days 14 ``` The backup runs `mongodump --archive` and reads its output as a stream. As the stream passes, SafeGrd counts each collection's documents and checks the CRC `mongodump` writes for it. The recorded counts match what is in the backup, even while the database is being written to. ## The user to back up as Back up as a user with MongoDB's built-in `read` role on the database. That is enough to back up every collection with its indexes and views. Create the user in `admin`, so the connection string's `authSource=admin` finds it. In `mongosh`: ``` db.getSiblingDB("admin").createUser({user: "safegrd_backup", pwd: passwordPrompt(), roles: [{role: "read", db: "shop"}]}) ``` If the surface's user holds a role that can write, such as `readWrite`, `dbOwner` or `root`, `safegrd doctor` warns and prints the line above. ### The sandbox for Fire Drills A sandbox drill (`drill.sandbox_url` on the host, or `--sandbox-target`) loads the archive with `mongorestore` into a database you use for nothing else. It counts each collection there, then drops them. The database has to be empty before each drill. Give the sandbox user `readWrite` on it to load and drop the collections, and `dbAdmin` so the drill can validate them and build their indexes: ``` db.getSiblingDB("admin").createUser({user: "safegrd_sandbox", pwd: passwordPrompt(), roles: [{role: "readWrite", db: "shop_sandbox"}, {role: "dbAdmin", db: "shop_sandbox"}]}) ``` ## TLS and a provider's own CA Atlas signs with a public CA, so `mongodb+srv://` strings verify as they are. If your server's certificate is signed by a CA of its own, give SafeGrd that CA. On a host, add `?tls=true&tlsCAFile=/path/ca.pem` to the URL, or set `ca_file` on the surface in the config. For a database SafeGrd backs up for you, paste the CA certificate when you add it, or later with **CA certificate** on the surface. SafeGrd refuses a URL that also says `tlsInsecure=true` or `tls=false`. ## Incremental backups A backup uploads only the collections that changed since the last run, and only the parts of them that changed. Each backup is a run of the surface's repository, like a [PostgreSQL backup](https://safegrd.dev/docs/surfaces/postgres#incremental). SafeGrd splits the archive by collection. The header and every collection's options and indexes go in one small file. Each collection's documents go in a file of their own, in the order `mongodump` read them. A collection nobody wrote to since the last run has the same bytes, so it uploads nothing. A collection that changed uploads the 64 KiB chunks around the documents that moved. Every run reads the whole database and is a restore point. The snapshot detail in the console shows what each collection uploaded. Measured on a 290 MB database, a run with no writes uploaded 0.1 MB, and 2,000 documents appended to one collection uploaded 0.5 MB. Updating 400 documents across a collection of 300,000 uploaded 19 MB, the chunks around each one. An inserted document costs only itself, whatever its `_id`, because `mongodump` reads in storage order and a new document sits at the end. To take one archive per backup instead, use `--format tar`, or `format: tar` on a surface. When you add a surface in the console, **Backup type** sets this. Each backup is then `mongodump`'s own archive, which `mongorestore --archive` loads without SafeGrd once it is decrypted. To restore or drill a run, SafeGrd puts the archive back together, one block per collection, and hands it to `mongorestore`. For a run, `mongodump` reads one collection at a time. A whole run restores into an empty database: `safegrd restore --snapshot --target mongodb://…`. You can also restore one collection of a run into a database that has other collections, with `safegrd restore --snapshot --table orders --target mongodb://…`. SafeGrd hands `mongorestore` an archive that holds only that collection's documents, options and indexes. The collection has to be missing from the target, or empty. In the console, the Tables tab has **Restore collection** for this. A full backup (`tar`) restores only whole. ### What the host needs Install the MongoDB Database Tools (`mongodb-database-tools`) on the host, from MongoDB's apt or yum repository or its download page. Ubuntu and Debian do not ship them. To point SafeGrd at specific binaries, set `SAFEGRD_MONGODUMP` and `SAFEGRD_MONGORESTORE`. SafeGrd passes them the connection string in a private file only the owner can read, never on their command line. ### What a backup holds - Each collection is read consistently, but not every collection at the same moment, because `mongodump --oplog` works only for a whole server. A document written to one collection during the backup may be in it while a related document in another collection is not. A [point-in-time restore](https://safegrd.dev/docs/surfaces/mongodb#pitr) brings every collection to one moment. - The backup keeps indexes and view definitions. Users and roles are managed at the server level, so a single-database archive does not include them. - Each backup records every collection's document count, size and shape. The shape is the collection's top-level fields across every document, each with its BSON type (`mixed` where documents disagree). `_id` is recorded as the primary key, and a field every document carries as not null. The console's _Tables_ tab shows them. Threat Shield warns when a field is gone from every document in the next backup ([what each snapshot records](https://safegrd.dev/docs/threat-shield#record)). For a collection with more than 500 top-level field names, SafeGrd records the first 500 and no hash. - Restore into an empty database. It can have any name. The restore checks document counts against the snapshot manifest. If a restore fails partway, drop the target database before you run it again. ## Check the backups another tool makes If Coolify or a cron `mongodump` already backs up the database, SafeGrd can check those backups where that tool stores them instead of taking its own. Set it up as for [a PostgreSQL surface](https://safegrd.dev/docs/surfaces/postgres#watch), with `type: mongodb`: ``` surfaces: - id: shop-db type: mongodb schedule: daily watch: url: s3://my-backups/data/coolify/backups/databases/team-0/shop-db endpoint: https://ACCOUNT.r2.cloudflarestorage.com access_key: env:R2_KEY_ID secret_key: env:R2_SECRET ``` - The check reads `mongodump --archive` files, gzipped or not. Coolify's `mongo-dump-*.tar.gz` is one gzipped archive, not a tar. The check counts each collection's documents and checks each collection's CRC as mongodump wrote it. An archive that was cut short or damaged in the upload counts as a failed backup. An archive of the whole server is checked, but drilled in memory only, because one sandbox takes one database. - A Fire Drill loads the archive into the sandbox with `mongorestore`, renaming the archive's database to the sandbox's, and compares every collection's documents. mongorestore stops at the first error and writes as it goes, so a failed load leaves part of the archive in the sandbox. The drill empties it afterwards. `drill.sandbox_url` must name a MongoDB database. A sandbox of another engine is refused before anything connects. - Under one Coolify tree, a `type: mongodb` surface reads only the `mongo-dump-*` files. The PostgreSQL and MySQL backups beside them belong to surfaces of their own type. - If you have no host, add the surface in the console under _Back up on SafeGrd_ and tick _Check the backups another tool makes_, as [the PostgreSQL page](https://safegrd.dev/docs/surfaces/postgres#watch) describes. SafeGrd holds the bucket key and runs the checks and drills on machines it starts. ## Restore to a point in time You can restore the database to any second inside a window, 7 days by default. Add a `wal:` block to the surface, and the resident daemon follows the replica set's oplog. Without it, a backup restores the database as it was when the run started. ``` surfaces: - id: app type: mongodb database_url: mongodb://safegrd_backup:...@db.internal:27017/app?authSource=admin wal: {} ``` This needs a replica set, because only a replica set keeps an oplog. To run a single server as a replica set of one member, start `mongod` with `--replSet rs0` and run `rs.initiate()` once. A standalone server and a sharded cluster (`mongos`) are refused, and the daemon says why. The backup user reads the oplog with `read` on `local`, the narrowest built-in role that can: ``` db.getSiblingDB("admin").grantRolesToUser("safegrd_backup", [{role: "read", db: "local"}]) ``` The oplog belongs to the whole replica set, so with this role the user can read the changes to every database on it. The daemon keeps only this database's entries. - Every 5 seconds (`interval_seconds`), the daemon reads the oplog entries for this database since its last read. It compresses and encrypts them on the host and pushes them to your bucket unlocked. Once a day it joins them into one locked bundle and deletes the unlocked copies. On hosted storage, this comes with Growth (7 days) and Scale (30 days). - A base backup is a `mongodump` of the database, taken between two reads of the oplog's newest entry. A restore replays every entry after the first of those reads over it, which brings every collection to the same moment. The daemon takes a base when it starts following and every 7 days (`base_every_days`). To take one now, run `safegrd wal base --surface app`. - Only the resident daemon follows the oplog. When it starts again, it carries on where the archive ends. If the oplog has rolled past that point in the meantime, the daemon says so, starts again from the newest entry and takes a new base. The window then starts from there. Size the oplog to hold at least a day. `safegrd doctor` shows how much it holds. - Once a day the daemon records a restore point: every collection's document count, read at one moment of the oplog. Every 7 days it restores the archive into the surface's `drill.sandbox_url`, to the newest restore point and to the end, and checks the counts. To run that drill now, use `safegrd wal drill --surface app`. - On MongoDB Atlas free clusters, a user can read the oplog and nothing else in `local`. The daemon's query meets their rule that an oplog namespace match starts with `^`. To restore into an empty database: ``` safegrd wal restore --surface app --to "2026-10-17 14:52:00" --target-url "mongodb://restorer:...@db.internal:27017/app_restored?authSource=admin" ``` The surface's **Point in time** tab in the console writes this command for a moment you pick. The command loads the newest base backup from before that moment (UTC). It then replays this database's oplog entries up to that moment with `mongorestore --oplogReplay`. Oplog entries are stamped to the second, so an entry from the same second as the moment you give is not replayed. To restore to a restore point, use `--to-point `. To restore everything archived, use `--to latest`. The target database can have a different name. The user you restore with needs `readWrite` on it and the built-in `restore` role, which lets `mongorestore` apply oplog entries: ``` db.getSiblingDB("admin").createUser({user: "restorer", pwd: passwordPrompt(), roles: [{role: "readWrite", db: "app_restored"}, "restore"]}) ``` --- Source: https://safegrd.dev/docs/surfaces/sqlite # SQLite You can back up a SQLite file on the host with nothing else installed, including a database in WAL mode. This one command takes a backup now: ``` # a database file on this host; no client tools needed safegrd backup --database-url sqlite:///var/lib/app/app.db --retention-days 14 ``` To back up on a schedule and get Fire Drills, add the file as a surface in `~/.safegrd/config.yaml` and run the [daemon](https://safegrd.dev/docs/daemon). Pick one way per database. A surface starts its own history, so any backups you took earlier with `--database-url` show in the console as a separate surface. ``` # ~/.safegrd/config.yaml surfaces: - id: app-db type: sqlite database_url: sqlite:///var/lib/app/app.db schedule: 6h # check it, then run it as a service safegrd config validate && safegrd daemon install ``` If you copy the file off the disk while the app writes to it, the copy can be torn. In WAL mode the newest transactions are also still in the `-wal` file, and a copy of the database file misses them. SafeGrd copies the database through SQLite itself, inside one read transaction. The copy holds every committed transaction, including those still in the WAL, and your app keeps writing. [SQLite WAL mode and backups](https://safegrd.dev/guides/sqlite-wal-backup) shows the difference on a test database. The SQLite engine is built into `safegrd`, so the host does not need `sqlite3`. ## What each backup does Each backup is a run of the surface's repository, as with PostgreSQL. The copy keeps every page where the database has it, so a run uploads and locks only the 64 KiB chunks whose pages changed. On a 228 MB database with 400 rows updated since the last run, a backup uploaded 6.8 MB. Inserts into an index on random keys, such as UUIDs or email addresses, touch pages all over that index and upload more. For one archive per backup instead, use `--format tar`, or `format: tar` on a surface. The archive is compacted with `VACUUM INTO`. Once you decrypt and unpack it, any SQLite client can open the file. - In WAL mode a backup does not block writers. In rollback-journal mode, writers wait while the copy is taken. Run `safegrd doctor` to see which mode the database is in, and `PRAGMA journal_mode=WAL;` to switch it. - Before any of the copy is uploaded, it goes through `PRAGMA integrity_check`. A damaged database is not stored, and the backup tells you why. Earlier snapshots are not touched. - The copy is written to an owner-only directory under the system temporary directory and deleted after the upload. It needs about as much free space as the database. Set `TMPDIR` to use another disk. ## Restore A restore writes a new file and does not overwrite an existing one: `safegrd restore --snapshot --target sqlite:///var/lib/app/restored.db`. Neither the file nor its `-wal` may exist yet. SafeGrd checks the copy with `PRAGMA integrity_check` and then renames it into place. `--to-sql` works for PostgreSQL only. To put the restored file in place of the live one: - Stop the app. - Move the old `app.db`, and any `app.db-wal` and `app.db-shm`, aside. - Move `restored.db` to `app.db`. The restored file is in rollback-journal mode. If the app expects WAL, run `PRAGMA journal_mode=WAL;` on it first. - Start the app. You can also restore one table into a database file that has other tables: `safegrd restore --snapshot --table accounts --target sqlite:///var/lib/app/app.db`. Stop the app first, so nothing writes while the rows land. In the console, the Tables tab has **Restore table** for the same thing. SafeGrd stages the backup beside the file and attaches it, so it needs about as much free space as the database. If the table is missing, SafeGrd creates it as the backup defines it, with its indexes and triggers. If it exists, it has to be empty and have the same columns. The rows are copied over in one transaction. ## Restore to a point in time A backup restores the database as it was when the run started. Add a `wal:` block to the surface and the resident daemon also follows the database's WAL. You can then restore to within a second of any moment in a window, 7 days by default: ``` surfaces: - id: app-db type: sqlite database_url: sqlite:///var/lib/app/app.db wal: {} ``` Every second, the daemon copies the transactions committed since the last copy to your bucket, compressed and encrypted on the host and unlocked. Once a day it joins them into one locked bundle and deletes the unlocked copies, as it does for PostgreSQL. On hosted storage, this comes with Growth (7 days) and Scale (30 days). ### What it needs - The database has to be in WAL mode (`PRAGMA journal_mode=WAL;`). - The daemon needs write access to the database and its directory. It adds one table, `_safegrd_wal`. It also keeps a read transaction open, so the WAL cannot be reset before it is copied. - Only the resident daemon follows the WAL. `safegrd daemon run --once` from cron cannot, and tells you so. The daemon checkpoints the database itself when the WAL passes 4 MB (`checkpoint_mb`). While it follows the WAL, your app's own `wal_checkpoint(TRUNCATE)` or `RESTART`, and `VACUUM`, wait for it and may return `SQLITE_BUSY`. Leave checkpointing to SQLite's default or to the daemon. Each time the daemon starts, it takes a copy of the database and begins a new run of WAL from it. It does the same every 7 days (`base_every_days`). ### How the daemon checks it Once a day the daemon records a restore point with each table's row count. It counts under the write lock, so the counts are exact. Every 7 days it restores the archive on the host, to the newest restore point and to the end, checks the counts and runs `PRAGMA integrity_check`. To run this check now, use `safegrd wal drill --surface app-db`. ### Restore to a moment Restore into a new file: ``` safegrd wal restore --surface app-db --to "2026-10-17 14:52:00" --out /var/lib/app/restored.db ``` In the console, the surface's **Point in time** tab writes this command for a moment you pick. The file holds every transaction committed up to the last copy at or before that moment (UTC). It passes `PRAGMA integrity_check` before it is renamed into place. - `--to-point ` restores to a restore point instead. - `--to latest` restores every segment in the archive. - If a segment is missing before the moment you asked for, the restore is refused and names the segment. - `safegrd wal status --surface app-db` shows the window. --- Source: https://safegrd.dev/docs/surfaces/files # Files and directory trees You can back up any directory tree on a host. Each run uploads only what changed, and you can restore any version of any file on its own. ``` # a directory tree, incremental safegrd backup --files /srv/uploads # with exclusions safegrd backup --files /srv/app --exclude '*.tmp,node_modules/*' # one archive per backup instead safegrd backup --files /srv/app --format tar ``` SafeGrd backs up a symlink as a symlink and does not follow it. A link that points outside the tree cannot pull the rest of the filesystem into your backup. FIFOs, sockets and device nodes are left out, and the backup lists each one. ## Incremental backups SafeGrd cuts each file into chunks of about 1 MiB, by content. A one-byte edit stores one chunk, and an insert does not shift the chunks after it. Chunks are compressed, encrypted on the host and gathered into packs of about 32 MiB. The packs are the objects in storage. If a file's size, times and inode have not changed since the last run, SafeGrd does not read it again. Every seventh run reads every file anyway. To do that now, pass `--rescan`. Runs are grouped into monthly _epochs_. The first run of each UTC month uploads every file once. Every later run that month uploads only the chunks the month does not already hold, plus a few kilobytes of metadata. Months share nothing, so each object is locked once, when it is written, and no lock is ever extended. A new epoch also starts when a host's cache is gone (a rebuilt host), when you increase retention, or when you pass `--new-epoch`. ``` # a VM's root, in ~/.safegrd/config.yaml, run by the daemon surfaces: - id: vm-root type: files roots: ["/"] excludes: ["/home/*/.cache"] schedule: "@daily" ``` A root of `/` stays on one filesystem and leaves out `/proc`, `/sys`, `/dev`, `/run`, `/tmp`, `/var/tmp`, swap files and SafeGrd's own cache. A surface can list several roots, as long as none is inside another. Set exclusions with the surface's `excludes`, or with _Leave out_ in the console. Each run tells you how many entries were left out and which patterns matched them. ## Back up a surface now ``` safegrd backup --surface vm-root ``` A surface backs up on its schedule. To take a backup now, run `backup --surface`. It goes into the same history the daemon writes. `backup --files` backs up a directory that is not a surface, as a separate history. ## Find and restore one file ``` safegrd find etc/nginx/nginx.conf safegrd restore --path etc/nginx/nginx.conf --target-dir ./out safegrd restore --path etc/nginx/nginx.conf --version 2 --target-dir ./out safegrd restore --snapshot --path 'var/www/**' --target-dir ./out safegrd find --deleted var/www/uploads/ ``` `find` lists every kept version of a file. A version is one content. It was first seen in the snapshot that first held those bytes, and last seen in the last snapshot that did. So a file that stayed the same for months is one version. A change of permissions alone does not make a new version. When you restore a version, it gets that snapshot's permissions. ``` etc/nginx/nginx.conf (vm-root) # FIRST SEEN LAST SEEN SIZE SHA-256 SNAPSHOTS 3 2026-10-02 02:00 (current) 2.1 KiB 9f1c2a7b0d3e 2 2 2026-09-14 02:00 2026-10-01 02:00 2.0 KiB 4b8e11c0aa52 18 (2 epochs) 1 2026-09-01 02:00 2026-09-13 02:00 1.9 KiB 77d0e3f5b2c1 13 ``` Versions are numbered from the oldest and printed newest first. Pass the number to `restore --version`. With one `--path` and no `--version` or `--snapshot`, restore takes the newest version. Each snapshot is a point in time you can restore to, so the surface's schedule decides how far apart they are. A daily surface gives you one point per day. To get a file as it was on a given day, pick the version whose first-seen and last-seen dates bracket that day. Or pass that day's snapshot id with `--path`. - A directory pattern selects everything below it. `*`, `?` and `**` work as wildcards. - `--deleted` lists only the files the newest snapshot no longer holds, with the last snapshot that held each one. - `--json` prints the same histories for scripts. Paths you give to `find` and `--path` are relative to `/`. A snapshot of one directory restores that directory's contents into the target, the same as an archive of it does. A snapshot of `/`, or of several roots, restores each entry at its path from `/`, under the target. Restoring one file reads that file's chunks and the snapshot's metadata, and nothing else from the tree. `find` and every restore run on the host that has the private key, and decrypt there. [Runbook 5](https://safegrd.dev/docs/runbooks#one-file) walks through it on a recovery machine. If the host's organization has other hosts, `find` searches the surfaces this host has the key for, and tells you how many others it did not search. `--surface ` searches one surface. With a SafeGrd-managed key, it fetches that surface's key for the search. ## What is locked, and for how long Each epoch gets two lock dates when it opens. Let T be the longest of `retention_days`, `keep_daily` and `keep_weekly`, in days. - Objects written by the month's first run are locked until the end of the month plus T, plus 3 days. If the first run's own retention (the monthly copy, with `keep_monthly`) ends later, that date plus 3 days applies instead. - Objects written by every later run are locked until the end of the month plus T. A snapshot is kept at most as long as the objects it uses. So when a backup prints _Locked until_ and a date, every byte it needs is locked until that date. Once the later objects expire, the month's first snapshot (the monthly copy) still restores from its own objects. ## What it holds Storage holds the kept monthly copies and about two months of changes. At the turn of each month it also holds one more full upload, while last month's objects are still locked. A 200 GB tree with 2% daily change and 30-day retention holds about 500 to 650 GB on average. Daily archives of the same tree would hold about 6 TB. On hosted storage, the console warns you a week before the month turns if that upload will take you past 80% of your plan. ## What the storage's holder can see File names, contents, directory structure and which content repeats are encrypted on your host. Whoever holds the storage (you, or SafeGrd on hosted storage) can see the size of each pack and how many packs each run wrote. That shows how much changed. A single-archive backup shows only each archive's size. ## Check a repository ``` safegrd check --all --read-data ``` `check` confirms three things for each snapshot: - Every pack it names is in storage and agrees with its index. - Every directory listing decodes. - The content digest, recomputed from those, matches the one recorded at backup time. `--read-data` also opens every chunk. ## List, copy and prune `safegrd list` shows incremental snapshots in their own table. For each one it gives the month (epoch) it belongs to, whether its run opened the month or followed it, the size of the whole tree it holds, and the bytes its run uploaded. `safegrd export` copies a repository a whole month at a time, because a snapshot needs every object of its month. `restore --from` reads the copy. In your own bucket, `safegrd prune` deletes a month's expired objects once every snapshot that could use them is past its retention. It never deletes from the month that holds the surface's newest snapshot. Restoring an incremental snapshot needs CLI v0.0.10 or later. An older CLI does not see these snapshots. Run `safegrd --version` to see which you have. ## What a restore brings back This applies to CLI releases after v0.0.2. Earlier releases restore every directory as 0755 and do not refuse a symlink in the target. Run `safegrd --version` to see which you have. | Property | Restored? | | --- | --- | | Contents | Yes. Every regular file is checked against the digest sealed at backup time. | | Permissions | Yes, for files and directories, whatever the restoring shell's umask. A `0700` directory comes back `0700`. | | Modification times | Yes, for files, directories and symlinks. Restoring a one-directory snapshot into a new target gives the target that directory's mode and time. | | Symlinks | Yes, as symlinks pointing where they pointed. | | Ownership | Only when `safegrd restore` runs as root. Otherwise files belong to whoever ran it, and the restore tells you. Snapshots taken before 24 September 2026 did not record owners. | | Hard links | No. Each linked name comes back as its own copy, so a tree full of hard links restores larger than it was. | | Extended attributes, ACLs, SELinux labels, capabilities | No. If a tree depends on them (an SELinux-labeled web root, a binary granted `setcap`), reapply them after a restore. | | setuid, setgid and sticky bits | Incremental backups: the sticky bit always, setuid and setgid when `safegrd restore` runs as root and puts the owner back. A `1777` directory comes back `1777`. Backups with `format: tar`: no, only the permission bits. | | Sparse files | Contents yes, sparseness no. Holes are written out in full. | | FIFOs, sockets, device nodes | Not backed up. The backup lists each one it leaves out. | When it finishes, `safegrd restore` prints what it restored and what it could not: ownership when it did not run as root, and the properties above that are never restored, so you know what to reapply. The target must be empty or not exist yet. A restore first writes everything into a staging directory inside the target and checks every file against its SHA-256. Only when every file matches does it move them into place. If one does not match, nothing is left in the target, and the restore names the file. ## What a Fire Drill checks A drill restores an incremental snapshot whole into a scratch directory on the host, and checks every file against its SHA-256. It then reads the restored files back, recomputes the snapshot's content digest from them, and compares it with the digest SafeGrd recorded at backup time. The scratch directory is removed afterwards. A single archive is checked in memory, and nothing is written to disk. Every entry has to match the manifest sealed inside the archive: names, types, symlink targets, the file count and byte volume, and a SHA-256 for every file. To rehearse your team's own restore procedure, see the [recovery runbooks](https://safegrd.dev/docs/runbooks). ## A bucket as a surface A files surface can read a bucket over the S3 API instead of a directory on the host. Use it for Supabase Storage, whose own backups leave out the uploaded files, or for any S3-compatible store. Every object under the prefix is backed up as a file at its key. The content type the store reported is sealed in the manifest. The archive restores into a directory like any other files snapshot, or back into a bucket, and a Fire Drill checks it the same way. The keys reach every bucket the store has, so pass them as `env:VAR` or `file:/path`. ``` # Supabase Storage, one bucket, one backup safegrd backup --files s3://avatars \ --source-endpoint https://PROJECT.storage.supabase.co/storage/v1/s3 \ --source-region eu-central-1 \ --source-access-key env:SUPABASE_S3_KEY --source-secret-key env:SUPABASE_S3_SECRET # as a surface the daemon backs up on its schedule surfaces: - id: avatars type: files roots: ["s3://avatars"] object_store: endpoint: https://PROJECT.storage.supabase.co/storage/v1/s3 region: eu-central-1 access_key_id: env:SUPABASE_S3_KEY secret_access_key: env:SUPABASE_S3_SECRET ``` A bucket is backed up as one archive per backup, because the incremental repository walks a filesystem. Each backup reads every object twice: once to hash it for the manifest and once to stream it. A store that meters egress bills two reads of the prefix per run. On Supabase, the endpoint and region are on the project's Storage settings page, under S3 connection, and you create the keys there. A key reaches every bucket in the project and skips row-level security, so keep it on the host that backs up. Supabase's folders are `.emptyFolderPlaceholder` objects, and they are backed up like any file. Supabase lists what its `storage.objects` table holds. A row with no file behind it (written by SQL, or restored before its file was uploaded) answers 404. The backup stops at that row and names it, so it does not record a file it did not read. ### Restore into a bucket `restore --target s3://bucket/prefix` uploads a files snapshot into a bucket. It takes the same `--source-*` flags as the backup. Each file goes to `prefix/path` with the content type the backup recorded, so a PDF is served as `application/pdf` again. - Each file's SHA-256 is checked against the sealed manifest while it uploads. A file that differs is not written. - Objects already at a key are replaced. Restore into a new prefix first to compare, then into the original one. - A snapshot that holds symlinks is refused before anything is written, because a bucket has no object for a link. Restore that one with `--target-dir`. ``` safegrd restore --snapshot snap-20261012-030000-a1b2c3 --target s3://avatars \ --source-endpoint https://PROJECT.storage.supabase.co/storage/v1/s3 \ --source-region eu-central-1 \ --source-access-key env:SUPABASE_S3_KEY --source-secret-key env:SUPABASE_S3_SECRET ``` ### A bucket SafeGrd backs up If you have no server of your own, add the bucket in the console: _Add surface_, _Back up on SafeGrd_, then _A bucket (Supabase Storage, S3)_. Give it the endpoint, bucket, prefix, region and an S3 key. A bucket counts as one surface. SafeGrd holds this bucket's key, encrypted, and uses it only on the machine it starts for each backup. Every use is in the key access log. The machine reads the bucket the way the CLI does above and writes one archive per backup to hosted storage. Drills read it back on your plan's schedule. When you add the bucket, SafeGrd's server lists it once, so a refused key shows up then. To replace the key, use _Credential_ on the surface. If you added a Supabase project with its sign-in, its Storage needs no key. SafeGrd backs up [every bucket from one sign-in](https://safegrd.dev/docs/hosted#supabase), as one surface. ## One archive per backup To store each backup as one encrypted tar, use `--format tar`, or `format: tar` on a surface. The manifest is sealed inside the tar, so SafeGrd's servers never see your file names. A tar backup takes one root. Every backup uploads the whole tree, and a restore reads the whole archive. Snapshots taken this way restore the same way whatever the default is. On hosted storage, you can download one from the console as a single file. --- Source: https://safegrd.dev/docs/surfaces/email # Email over IMAP You back up a mailbox over IMAP with one command, and restore it as `.eml` files that any mail client opens. ``` # any IMAP server that accepts a password: Gmail, Fastmail, self-hosted export SAFEGRD_EMAIL_PASSWORD='your-app-password' safegrd backup --email --email-host imap.example.com --email-user ops@example.com ``` SafeGrd fetches each message as RFC 5322 and streams it into the sealed archive, keeping the folder structure. It backs up every folder except spam and trash. To back up fewer, name them with `--email-folders`. Put the password in `SAFEGRD_EMAIL_PASSWORD` instead of passing `--email-password`. A password given as a flag shows up in shell history and in the process list. ## Providers | Provider | IMAP server | Password | | --- | --- | --- | | Gmail and Google Workspace | `imap.gmail.com:993` | An app password, which you create at [myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords). The account needs 2-Step Verification. A Workspace administrator can turn app passwords off. | | Fastmail | `imap.fastmail.com:993` | An app password with IMAP access: Settings, Privacy & Security, Manage app passwords and access. | | Microsoft 365 and Outlook.com | Not supported yet | Microsoft turned off password sign-in for IMAP, app passwords included, and requires OAuth. SafeGrd signs in with a password, so it cannot back up these mailboxes until it supports OAuth. | | Self-hosted (Dovecot, Cyrus and others) | Your server, port 993 | The mailbox's password. For a certificate from an internal CA, see [below](https://safegrd.dev/docs/surfaces/email#private-ca). | ## Gmail labels Gmail shows each label as a folder, so one message can appear in several folders. SafeGrd backs up only Gmail's All Mail folder and records each message's labels beside it. Each message is downloaded and stored once. Spam and Trash are left out. For this to work, All Mail has to be visible over IMAP. It is by default. To check, open the Labels tab in Gmail's settings and make sure **Show in IMAP** is ticked for All Mail. If it is off, the backup reads one folder per label instead. It stores a message once for each label it has, and prints a warning. If you list `folders` on the surface, SafeGrd backs up exactly the folders you named. ## Where the password comes from By default the password stays on the host. The surface's `credential` block says `from: env` with the variable's `name`, `from: command` with a command that prints the password, or `from: file`. SafeGrd learns the variable's name and never sees the password. To have SafeGrd hold the password instead, set `credential: {from: safegrd}` on the surface and name nothing on the host. SafeGrd keeps the password encrypted and gives it only to this surface's host when the mailbox is due. The daemon keeps it in memory for that run. Add it in the setup wizard, or open the surface under **Surfaces**, click **Edit**, then **Add credential**. SafeGrd refuses an incomplete block, such as `from: env` with no `name`, instead of guessing. ## A mailbox behind a private CA If your mail server's certificate chains to an internal CA, point the surface at a PEM bundle on the host with `ca_file: /etc/ssl/mail-ca.pem`. For a one-off backup, pass `--email-ca-file`. `SAFEGRD_EMAIL_CA_FILE` sets the bundle for every email surface on the host that does not name its own. SafeGrd adds the bundle to the system roots and still trusts them. You cannot turn certificate checks off, because the mailbox password travels over this connection. ## Restoring a mailbox ``` safegrd restore --snapshot --target-dir /restored/mailbox ``` You get one directory per folder, with each message as a standard RFC 5322 `.eml` file. Any mail client or forensic tool can open or import them. A Gmail mailbox comes back as one `All Mail` directory. Its labels are in `.safegrd-email-manifest.json` at the top of the target directory, which lists each message's `path` with its `labels`, such as `\Inbox` or `Work`. ## What a Fire Drill checks The drill checks that the message count matches the manifest and that every message's SHA-256 matches. It fails if the archive holds a message the manifest does not list, or if a message the manifest lists is missing. It also reads each message's headers as RFC 5322 mail and reports how many do not parse. Those messages were stored that way on the mail server, often old spam. They are restored byte for byte and do not fail the drill. The drill writes nothing to disk. --- Source: https://safegrd.dev/docs/surfaces/wordpress # WordPress sites You back up a WordPress site with the SafeGrd Backup plugin, from wp-admin. You install nothing else on the server, so it works on shared and managed hosting. Each backup is a complete snapshot of the database and the site's files, encrypted on your server before it leaves. Backups go to SafeGrd's hosted storage, locked so nobody can delete them early. You can use [your own bucket](https://safegrd.dev/docs/surfaces/wordpress#own-bucket) instead, locked or not, as you set it up. A backup uploads only what changed since the last one. [Pricing](https://safegrd.dev/pricing) lists what each plan includes. ## Install and connect - In wp-admin, open Plugins, Add New, search for SafeGrd Backup, press Install Now, then Activate. The plugin is [safegrd-backup](https://wordpress.org/plugins/safegrd-backup/) in the WordPress.org directory, and its updates come from there too. If the site cannot reach WordPress.org, the same zip is on the [plugin's releases](https://github.com/safegrd/wordpress-plugin/releases). - Open Tools, SafeGrd, choose how your key is managed, and press Connect. In the tab that opens, sign in or create an account. Check that the tab shows the same code as wp-admin, then press Connect site. To connect without the tab, open Connect with a token instead and paste a token from Tokens in the console. The first backup starts right after you connect. After that, a backup runs once a day (or once a week, under Settings) through WP-Cron, which runs when someone visits the site. To run one straight away, press Back up now on the same page. ``` # the same from WP-CLI, with a token from Tokens in the console wp plugin install safegrd-backup --activate wp safegrd connect --token=env:SAFEGRD_TOKEN wp safegrd backup ``` ## How your key is managed You choose this when you connect. With a **SafeGrd-managed key**, the default, SafeGrd keeps your key sealed and releases it only to your enrolled hosts, so you can restore even after losing this site. This is also what lets SafeGrd test-restore your backups for you. With a **Customer-managed key**, only you can decrypt these backups. The plugin shows the key once, when you connect. Keep a copy somewhere safe, such as a password manager. The site keeps only the public half, so it can write backups but cannot read them. ## Back up to your own bucket You can keep a site's backups in your own S3 bucket instead of SafeGrd's, with Object Lock on or off. The site holds no storage key. SafeGrd signs each upload and read with the key it holds for the bucket. - In the console, press Storage on the site's project, choose Your own bucket, and enter the bucket, its endpoint, its access key and its secret key. SafeGrd holds the secret key you enter. - Press Test connection. The test has to pass with the key SafeGrd holds. It also has to find Object Lock on, unless you chose a bucket without it. - The site's next backup goes to the bucket, and SafeGrd test-restores it there. Tools, SafeGrd and `wp safegrd status` show the bucket's name. The first backup in the bucket uploads every file once. Each backup is locked for the project's retention, in the bucket's Object Lock mode. In a bucket you set up without Object Lock (`worm_mode: NONE`, such as DigitalOcean Spaces), backups are not locked, and Tools, SafeGrd shows them as Not locked. SafeGrd deletes nothing in your bucket. To remove old backups, use the bucket's lifecycle rules, or run `safegrd prune` on a host with the key. The backups are filed the same way the CLI files a host's, under `//repo/`. A host with the bucket's key can verify and restore them without SafeGrd: ``` safegrd verify --snapshot snap-20261019-093000-1a2b3c --s3-bucket my-backups --s3-prefix sites --s3-access-key AKIA... --s3-secret-key env:S3_SECRET ``` If SafeGrd does not hold the bucket's key, or the bucket has not passed a test with it, the site stays on hosted storage and Tools, SafeGrd says why. ## What is backed up | Part | What is in the snapshot | | --- | --- | | Database | Every table that starts with the site's `$table_prefix`, in the form `mysqldump` writes. The tables are read in one consistent snapshot, or [table by table](https://safegrd.dev/docs/surfaces/wordpress#hosting) on a host that stops the request first. If another WordPress install shares the database and its prefix starts with yours (`wp_shop_` beside `wp_`), its tables are left out and listed. Views and triggers are listed as left out too | | Files | `wp-config.php`, `.htaccess` and `wp-content`: uploads, themes and plugins | | Left out | WordPress core (reinstall the version the backup names), caches, `upgrade`, `debug.log`, and other backup plugins' archives. Symlinks and unreadable files are listed after each run | | Left out by you | Paths and names you set under Tools, SafeGrd, Settings, such as `wp-content/ewww` or `*.zip`. You cannot leave out uploads, because the test restore checks every attachment. Each run lists what it left out and how many matched | Under Settings you choose daily or weekly backups and the hour they start, in the site's timezone. WP-Cron starts each one at the first visit after that hour. Tools, SafeGrd, Logs keeps the output of the last 20 backups, restores and downloads. `wp-config.php` holds the database password and the site's salts. It is in the snapshot because a restore needs it, and it is encrypted with everything else. The plugin refuses Multisite networks. It also refuses a site whose media another plugin keeps in object storage (WP Offload Media, WP-Stateless, Media Cloud), because those files are not on the server for it to read. ## What the test restore checks SafeGrd test-restores the newest backup on its own machines, in memory, on every plan. The drill fails if any of these fails: - Every pack the snapshot names is in storage and opens with the key. The snapshot's content root matches the one recorded when it was taken. - The database dump is complete, and every table has the rows the backup counted. - Every file matches the size and SHA-256 listed beside it, and no file is unlisted. - Every attachment the database names (`_wp_attached_file`) is a file in the snapshot. The list comes from the database half of the backup, so a missing upload fails the drill by name. ## Restore You can restore from wp-admin on any host. Install WordPress, install the plugin, and connect it to the same account. Tools, SafeGrd, Restore & download lists this site's backups. The _Site_ filter shows the backups of any other WordPress site in the account. _Open_ a backup to see what it holds: each part's size, the WordPress and PHP versions, and its plugins, themes and tables. Tick what to restore, press _Review restore_ to see what will be replaced, then press _Restore now_. You can also restore a site over itself from the same page. - The site keeps running on its own database and files while the restore loads and checks everything: every table's rows against the backup, every file against its SHA-256. Then one step swaps the restored ones in. - The restore keeps the tables and files it replaced. _Put the copy back_ undoes the restore by swapping them with what it put in. The restored version is then the copy kept aside, so pressing the button again restores it again. _Delete the copy_ frees the space. Copies kept by earlier restores are listed with their size. From the command line: `wp safegrd restore --undo`, `--delete-copy` or `--delete-older`. - The site keeps its own `wp-config.php` and domain. If the URL differs, the restore runs a search and replace from the old URL to the new one, serialized data included, and renames the tables to the new table prefix. - After the restore, the site's users are the ones in the backup. Sign in with an administrator account of the restored site. The site stays connected to SafeGrd. - You can restore only some parts: the database, plugins, themes, uploads, and the rest of wp-content. If you leave out the database, the site keeps its content, settings and sign-in. Plugin settings stay as they are too, because they live in the database. The SafeGrd plugin itself always stays at the version that is running. - You can also restore single plugins, themes or tables. _Some plugins only_, _Some themes only_ and _Some tables only_ list what the backup holds. For each plugin and theme, they show the version in the backup and the one installed now. Each one you tick replaces its own directory or table, and everything else stays as it is. Use this to roll back one plugin after an update, or to bring back the posts table alone. Restoring the users table brings back the backup's users with it. From the command line: `wp safegrd contents ` lists them, then `wp safegrd restore --plugins=akismet` or `--tables=wp_posts,wp_postmeta`. - While updates are waiting, the Updates, Plugins and Themes screens in wp-admin show how old the last backup is, next to Back up now. - A restore runs in slices, like a backup, so a host that stops long requests does not stop it. From the command line: `wp safegrd snapshots`, then `wp safegrd restore `, or `wp safegrd restore --only=plugins,themes`. ### Download a backup You can download one part of a backup as a file from the same page. The database comes as a gzipped SQL dump. Plugins, themes, uploads or the rest of wp-content come as a gzipped tar, and the tar also carries `wp-config.php` and `.htaccess`. The plugin writes the file on the site's server, decrypted, in slices, and an administrator fetches it from wp-admin. It is deleted from the server a day later. From the command line: `wp safegrd download uploads --to=uploads.tar.gz`, or for single tables, plugins or themes, `wp safegrd download database --tables=wp_posts --to=posts.sql.gz`. To restore a backup taken with a customer-managed key, open it on the same page and paste the key, or pass `--key=file:/path/to/key` to `wp safegrd restore`. The site checks that the key opens the backup. It keeps the key sealed until the restore swaps in, or for an hour if no restore starts, then deletes it. To restore onto a server without WordPress, use the [safegrd CLI](https://safegrd.dev/docs/install) on any host enrolled in the same organization. You need an empty MySQL or MariaDB database, an empty directory, and the `mysql` client. ``` # find the snapshot safegrd list # database into an empty database, files into an empty directory safegrd restore --snapshot snap-20261006-120000-a1b2c3 \ --target mysql://wp:...@db.example.com:3306/wordpress --target-dir ./restored ``` The restore prints the snapshot's WordPress version and site URL. Then rebuild the site in the directory your web server serves, here `/var/www/html`: ``` # 1. WordPress core, at the version the restore printed wp core download --version=6.8.3 --path=/var/www/html # 2. the restored files over it cp -a ./restored/. /var/www/html/ # 3. owned by the web server's user (www-data on Debian and Ubuntu) chown -R www-data:www-data /var/www/html # 4. wp-config.php pointed at the restored database wp config set DB_NAME wordpress --path=/var/www/html wp config set DB_USER wp --path=/var/www/html wp config set DB_PASSWORD '...' --path=/var/www/html wp config set DB_HOST db.example.com:3306 --path=/var/www/html # 5. only on another domain: search and replace the old URL, serialized data included wp search-replace 'https://old.example' 'https://new.example' --all-tables --skip-columns=guid --path=/var/www/html ``` If the site comes back at the same address, skip step 5. Otherwise run it with `--dry-run` first to see how many values it changes. Do not replace the URL as plain text in the SQL. That breaks serialized options, such as widget settings, because their recorded lengths no longer match. The restored site is not connected to SafeGrd, because every backup leaves out the plugin's own settings. Connect it again from Tools, SafeGrd. ## What each backup uploads A backup uploads only the chunks this month's repository does not hold yet. Files are cut into 4 MiB chunks, encrypted and gathered into packs. Each table is its own part of the dump, so an unchanged image, theme or table costs nothing. The first backup of each month uploads everything once, so every month stands on its own. Every snapshot is complete, and a restore never needs an earlier one. ## Large sites on strict hosts The plugin runs a backup in slices of a few seconds, each its own request. A slice lasts a third of the host's `max_execution_time`, at most 25 seconds. Each slice uploads what it read, saves where it got to, and starts the next one. If the host stops a request, only that slice stops, and the next one resumes from the last saved point. A file too large to read in one slice, such as a video, is stored 4 MiB at a time across as many slices as it needs. On a host that stops requests at 30 seconds, a 2 GB file backs up and restores this way. The plugin first reads the whole database in one request, as one consistent snapshot. On a host that stops requests at 30 seconds, a dump of about 1 GB fit only on a fast machine with MySQL's cache warm. If the host stops that request, the plugin reads the database table by table across slices instead. Each slice reads its tables in a consistent snapshot of its own. It reads a large table a page of rows at a time, by primary key rather than by position, so rows added or deleted in the meantime do not shift the pages. A table with no primary key is read whole in one slice. That backup still holds every table, but the tables were read at different moments, a few seconds to minutes apart. The plugin says so in its log and on Tools, SafeGrd. The snapshot in the console says so too, with the time span. The backup's manifest lists what each request read and when. To get one consistent snapshot of a large site, run backups from the server's cron, where WP-CLI has no time limit. Then turn off WP-Cron's own trigger with `define('DISABLE_WP_CRON', true);`: ``` 17 3 * * * cd /var/www/html && wp safegrd backup --quiet ``` ### Requirements The plugin needs PHP 7.4 or newer with the sodium, zlib and mysqli extensions, which PHP includes by default. To use a self-hosted SafeGrd server, set `define('SAFEGRD_SERVER_URL', 'https://...');`. For a private CA, set `define('SAFEGRD_CA_FILE', '/path/to/ca.pem');`. --- Source: https://safegrd.dev/docs/surfaces/laravel # Laravel You back up a Laravel app as one surface. Its database, its `.env` and `storage/app` go into one snapshot and get one drill result. The CLI runs on the app's host and reads the connection from `.env` at every backup, so you change nothing when the password rotates. ## Set it up ### From the app, with Composer Use this on a host where the app's user has no root, such as Forge, Ploi or shared hosting. It needs Laravel 10 or later and PHP 8.1 or later. ``` composer require safegrd/laravel php artisan safegrd:install ``` `safegrd:install` downloads the CLI, checks it against the release's `checksums.txt` and prints a code for you to approve in the console. Then it adds the app and takes the first backup. From then on, Laravel's scheduler runs `php artisan safegrd:run` every five minutes. That command backs up when the app is due and drills when a drill is due, so the host needs the scheduler's cron entry. The package gives the CLI the connection as Laravel resolved it, so a password set in the host's environment instead of `.env` works too. The package's options are in its [README](https://github.com/safegrd/laravel-package). Run `safegrd:install` as the user the scheduler runs as, which is the owner of `storage/`. To check the setup, run `php artisan safegrd:doctor`. A failed backup exits non-zero, writes an error to the app's log and dispatches a `SafeGrd\Laravel\Events\BackupFailed` event. ### On the host, with the CLI Use this on a server you manage. Enroll first, so the host's key is made with the custody you choose. Then add the app, running as the user the app runs as: ``` curl -fsSL https://safegrd.dev/install.sh | sh safegrd init --laravel /var/www/shop safegrd doctor sudo safegrd daemon install ``` `init --laravel` checks the app, tells you which connection it found, and adds the surface to the config. The surface id is `APP_NAME` as a slug, unless you set one with `--surface-id`. The daemon then backs up and drills the surface on its schedule. If you would rather use Laravel's scheduler than a service, `init` prints the line to add to `routes/console.php`. ``` # what init --laravel writes to ~/.safegrd/config.yaml surfaces: - id: shop type: laravel name: Shop roots: [/var/www/shop] ``` `roots` is the app's root directory, where `artisan` and `.env` are. Run the CLI as a user that can read `.env` and `storage/app`. That is usually the user PHP runs as: `forge` on a Forge server, `www-data` on Debian. Laravel Vapor and Laravel Cloud have no server for the CLI to run on. ## What is captured | Part | How | | --- | --- | | The database | Backed up first, the same way as that engine's own surface: `mysqldump` for MySQL and MariaDB, `pg_dump` and binary `COPY` for PostgreSQL, a page copy for SQLite | | `.env` | Whole, `APP_KEY` included, sealed like everything else | | `storage/app` | Every file with its mode and time. A file that did not change since the last backup is not read or uploaded again | | `database/migrations` and `database/schema` | A few kilobytes. The drill compares them with the `migrations` table | The backup leaves out `vendor`, `node_modules`, `storage/framework`, `storage/logs`, `bootstrap/cache` and the rest of the code, which lives in git. Inside `storage/app` it skips `backup-temp`, `livewire-tmp` and the zip archives spatie/laravel-backup writes there. To leave out more, add paths from the app's root to the surface's `excludes`, such as `storage/app/exports`. ## Why the .env is in the backup Laravel encrypts cookies, signed URLs, `encrypted` casts and anything passed to `Crypt` with `APP_KEY`. If you restore the database without the key it was written with, you have those columns and nothing to read them with. The `.env` is encrypted on the host with the host's key before it is uploaded. With a SafeGrd-managed key, SafeGrd keeps that key sealed and releases it only to your enrolled hosts. With a customer-managed key, only you can decrypt the backup, so keep a copy of the key somewhere safe. ## Where the connection comes from The backup reads the connection from `.env` the way Laravel's own config does. It uses `DB_URL` (or `DATABASE_URL`) when it is set. Otherwise it uses `DB_CONNECTION` with `DB_HOST`, `DB_PORT`, `DB_DATABASE`, `DB_USERNAME` and `DB_PASSWORD`. The `mysql`, `mariadb`, `pgsql` and `sqlite` connections are backed up. A SQLite path is relative to the app's root. An unset `DB_DATABASE` means `database/database.sqlite`. To back up as a different role from the app's, such as a read-only one, or with a connection set outside `.env`, give the surface its own `database_url` or `credential` block. `init --laravel --database-url env:VAR` writes one. The `.env` is still backed up whole. If your migrations table has another name, set it in `migrations_table`. The password read from `.env` is never written to the config or the state directory, and never sent to SafeGrd. | The backup refuses | Why, and what to do | | --- | --- | | No `.env` | The `APP_KEY` would not be in the backup. Create one, or back up the database alone as a database surface | | No `DB_CONNECTION` | Laravel's default changed in version 11, so SafeGrd does not guess. Set it, or give the surface its own `database_url` | | `sqlsrv` | SQL Server is not backed up yet | | `DB_SOCKET` | The CLI connects over TCP. Set `database_url` on the surface | With `FILESYSTEM_DISK=s3`, the uploads are in a bucket instead of `storage/app`. The backup still takes the database and the `.env`, and prints a warning. Back up the bucket as a [files surface](https://safegrd.dev/docs/surfaces/files) with an `object_store` block. ## What a Fire Drill checks The drill runs on every plan, in memory. A SQLite database is opened from a temporary copy, which is deleted afterwards. The drill checks that: - The database passes its engine's drill: the dump is complete and every table's rows match the manifest. - Every file reads back and matches its SHA-256. - The `.env` in the snapshot holds an `APP_KEY` of the length Laravel's cipher takes, 32 bytes for AES-256 or 16 for AES-128. The drill reports this check as `AppKey`. - The dump holds the `migrations` table with the rows the backup read, and at least one row for each migration file that had run. The drill reports this check as `Migrations`. If a migration had not run yet when the backup was taken, the backup's output names it. If any check fails, the drill fails. The drill does not boot the restored app yet. ## Restoring an app ``` # TARGET_URL names an empty database: mysql://, postgres:// or sqlite:///path/new.sqlite safegrd restore --snapshot --target env:TARGET_URL --target-dir /tmp/shop-restored ``` The restore loads the database into an empty database of the same engine. It writes the `.env`, `storage/app` and the migration files to an empty directory. Then check out the app at the version the restore prints, and in that checkout: - Copy `.env` and `storage/app` from the restored directory into it. - Point `DB_HOST`, `DB_DATABASE`, `DB_USERNAME` and `DB_PASSWORD` at the restored database. - Run `php artisan config:clear`, `php artisan storage:link` and `php artisan migrate:status`. --- Source: https://safegrd.dev/docs/storage # Storage and retention Your backups go to a bucket you own or, if your plan includes it, to SafeGrd's own locked bucket ([hosted storage](https://safegrd.dev/docs/storage#hosted)). Either way, each backup is encrypted on your host before it leaves. Who can decrypt it depends on how your key is managed ([Security and key custody](https://safegrd.dev/docs/security)). A bucket of your own has no per-gigabyte surcharge, and it stays in your account. ## Choose a sink | Sink | Notes | | --- | --- | | hosted | SafeGrd's own bucket, locked in compliance mode, with your plan's included storage. You have no bucket to set up. See [Hosted storage](https://safegrd.dev/docs/storage#hosted). | | local | A directory on the host. A filesystem-level lock stands in for Object Lock. Use it to try SafeGrd or on an air-gapped host. If the machine is lost, so are the backups. | | AWS S3 | The reference target. Nobody can shorten or remove a compliance-mode lock on AWS, and SafeGrd's retention depends on that. | | MinIO | Self-hosted and S3-compatible, with real Object Lock. Set `endpoint`. | | Cloudflare R2 | S3-compatible. Set `endpoint`. It has **no S3 Object Lock**, so set `worm_mode: "NONE"` ([details](https://safegrd.dev/docs/storage#cloudflare-r2)). | | Backblaze B2, Wasabi | S3-compatible, each with its own object lock features. Check the bucket with `safegrd doctor`. | | DigitalOcean Spaces | S3-compatible, but it **has no Object Lock at all**. You have to opt out of locking explicitly ([details](https://safegrd.dev/docs/storage#spaces)). | | SFTP | A directory on an SSH server. It has **no Object Lock**, and it accepts `worm_mode: "NONE"` and nothing else ([details](https://safegrd.dev/docs/storage#sftp)). | Set the sink once in `config.yaml`, or override it for one run with `--s3-bucket`, `--s3-endpoint`, `--s3-region` and `--s3-prefix`. You can also set a sink on a project, and every host in the project uses it. When more than one is set, the per-run flags win over the project's sink, and the project's sink wins over `config.yaml`. ## Hosted storage Set `storage.type: "hosted"` and you have no bucket to set up. The host never holds a storage credential. Each time it uploads a part or reads an object, it asks SafeGrd's server for a URL signed for that one request: this part, of this many bytes, or this version of this snapshot. The bytes go straight to SafeGrd's bucket. A signed URL cannot delete, cannot be used for another object, and expires in minutes. The server locks each snapshot in compliance mode, as in a bucket of your own, and the data is still encrypted on your host. ``` safegrd init --storage hosted --database-url env:DATABASE_URL safegrd enroll --token safegrd backup ``` Hosted storage is activated once your owner email is verified, so verify it before you run the first backup. ### Included storage and overage Your plan's included storage is counted in locked bytes. On the free plan, every byte kept counts, locked or not. You cannot lift a lock to make room. `safegrd doctor` shows how much is locked, and the console's Billing tab shows it against your included storage. On a paid plan, a backup that goes past your included storage is still accepted. Once a day the server samples what you hold, and it averages the samples over the billing period. At the end of the period it charges for the part above your included storage, per GB-month, rounded down to whole GB. The price is on the [pricing page](https://safegrd.dev/pricing). You get an alert at 80% and another when you pass your included storage. The console shows this period's overage so far, in dollars. Locked bytes count until their lock ends. If you lower a retention or move to a smaller plan, the locks already written keep their dates, and overage on those bytes continues until each lock ends. The console shows the date the last one ends. On the free plan and during the trial, there is no subscription to bill. A backup that would pass your included storage is refused with the reason, and you get an alert. If a subscription lapses, you can go a little past the last paid plan's included storage (the [terms](https://safegrd.dev/terms) give the margin). After that, backups are refused. Nothing already stored is affected, and you can always restore. ### How long hosted backups are kept If your config sets no retention, the plan's applies. Every backup gets a short lock, and daily, weekly and monthly copies are kept longer. The lengths are sized so a typical estate fits in the plan's included storage. You can set your own `retention_days` and `keep_*` within the plan's range. The server sets every lock, and never longer than the plan's longest tier. On the free plan, a backup is kept for 1 to 7 days (7 unless you choose otherwise) and locked for its first day. After that day it stays in the bucket unlocked until its retention ends, and it counts against your included storage until it is deleted. Each backup is a full copy, so each one can be deleted on its own date. A snapshot is deleted when its retention ends, which on a paid plan is when its lock expires. SafeGrd removes the object from the bucket within 24 hours after that. While a snapshot is locked, nobody can change or delete it. Its verification records and Fire Drill certificates stay in the console, marked expired. To keep snapshots longer, export them to your own bucket before the lock expires. ### Download a backup You can download up to four times your included storage a day, or as much as you hold if that is more. The limit covers every restore and Fire Drill, and a full `safegrd export` to your own bucket. To download a full backup, open its details in the console. Click _Download backup_, _Download metadata_ or _Download RECOVERY.md_ (the [recovery document](https://safegrd.dev/docs/storage#recovery-doc)). The backup arrives encrypted, straight from the bucket. Save the backup and its metadata in one folder and run `safegrd restore --from .safegrd --target ` (or `--target-dir` for files and mail). If the host has a SafeGrd-managed key, run the restore on one of your enrolled hosts, because SafeGrd releases the key only to them. An incremental snapshot has no backup file to download. Files, PostgreSQL and SQLite back up incrementally unless the surface is set to Full. Each run is stored as chunks it shares with the other runs of its month. For these, the console offers the metadata and RECOVERY.md. Restore one on a host with `safegrd restore --snapshot `, or copy it to your own bucket or a directory with `safegrd export`. MySQL and MongoDB backups are always full. ### Move to your own bucket You can move whenever you like. `safegrd export --to-bucket ` copies every snapshot across, still encrypted and still locked. Then switch `storage.type`. ### WordPress sites and databases SafeGrd backs up, in your own bucket A WordPress site, or a PostgreSQL database that SafeGrd backs up without a host of yours, can back up to the project's own bucket. SafeGrd holds that bucket's key and signs each upload and read with it, the same way it signs them for hosted storage with its own key. Neither the site nor the database backup is ever given the key. All of these have to be true: - You set up the bucket in the console, and SafeGrd holds its key. A bucket that a host registered from its own config does not count. - _Test_ passed with that key and found Object Lock on, unless you saved the bucket without it. - For a database SafeGrd backs up, the lock mode is Compliance or Governance. A WordPress site can also use a bucket you set up with `worm_mode: "NONE"` ([no Object Lock](https://safegrd.dev/docs/storage#spaces)). Until then, a site backs up to hosted storage. The surface's panel in the console, the plugin's Tools page and `wp safegrd status` tell you which condition is missing. You can add a database SafeGrd backs up to such a project only once all of them are true, and only as an incremental backup. A full backup is one archive, and SafeGrd writes those only to hosted storage. MySQL and MongoDB are added to a project on hosted storage. In your bucket, each backup is locked for the project's retention, in the bucket's lock mode. In a bucket without Object Lock, a site's backups are not locked, and the plugin and the console show them as _Not locked_. SafeGrd reads backups by name, so the bucket does not need versioning. Hosted storage's included amount, plan retention and download limit do not apply in your bucket. SafeGrd deletes nothing there. Your bucket's lifecycle rule does, or `safegrd prune` on a host with the key. The backups are under `//repo/`, the same layout a host writes. An enrolled host with the bucket's key can run `safegrd verify` and `safegrd restore` on them straight from the bucket. When the project moves between hosted storage and its bucket, the next backup uploads everything once. Older backups stay where they were written. ## Storage forecast Open _Projects_ in the console and click _Storage_ on a project. It shows what the project's storage holds now, and what it will hold once every surface has run through its retention. On hosted storage, it also shows the date you will reach your plan's included storage. - _Held now_ is every backup still locked. On hosted storage, it adds up the sizes the bucket reported and leaves out uploads still in progress. For your own bucket, it adds up the encrypted sizes your hosts reported, so it is an estimate. An incremental snapshot counts the bytes it wrote. - _Settles at_ multiplies what each surface writes per day by how long each backup is held, and adds up the surfaces. A surface that started recently is still filling up, so today it holds less than this. - _Full on_ is the first day the forecast passes your included storage. On a plan that bills overage, that is the day billing starts, and backups continue. The forecast replays the last 30 days. It assumes each backup written in that time is written again 30 days later, at the same size and with the same lock. What is already stored drops out on the day its lock ends. A surface that has backed up for less than 30 days is replayed over the time it has run. Because it replays what your hosts wrote, the forecast includes compression, the longer locks on weekly and monthly copies, and an incremental surface's larger first run each month. It does not predict growth. When a database grows, its backups grow, and the forecast follows. A surface with no backup yet is listed but not counted. On a bucket without Object Lock, a surface with no retention set is not counted either, because nothing removes its backups. To bring the figure down, shorten retention, back up less often, or move a surface to another project's storage. For your own bucket, the [Object Lock cost calculator](https://safegrd.dev/tools/object-lock-cost-calculator) turns a schedule and retention into a monthly bill. ## Setting up an AWS S3 bucket - Create the bucket **with Object Lock enabled**. You can only turn it on when you create the bucket. A bucket made without it cannot be converted, so you would have to replace it. ``` aws s3api create-bucket --bucket acme-safegrd --object-lock-enabled-for-bucket ``` Outside `us-east-1`, add `--create-bucket-configuration LocationConstraint=eu-west-1` (your region). - Create an access key with the policy below for each backup host, and one read-only key for recovery. - Run `safegrd doctor` on the host. It checks that the bucket answers and that Object Lock is on. The first `safegrd backup` proves the key can write. What a backup host needs. These are the S3 calls the CLI makes, and no others: ``` {"Version": "2012-10-17", "Statement": [ {"Effect": "Allow", "Action": ["s3:ListBucket", "s3:ListBucketVersions", "s3:GetBucketObjectLockConfiguration"], "Resource": "arn:aws:s3:::acme-safegrd"}, {"Effect": "Allow", "Action": ["s3:PutObject", "s3:PutObjectRetention", "s3:AbortMultipartUpload", "s3:GetObject", "s3:GetObjectVersion"], "Resource": "arn:aws:s3:::acme-safegrd/*"} ]} ``` That policy cannot delete anything. `s3:AbortMultipartUpload` only abandons an upload that failed halfway, so you are not billed for its parts. It cannot touch a finished object. A compromised host cannot hide a backup behind a delete marker either. The only SafeGrd command that deletes is `prune`, and it is off until you turn it on ([below](https://safegrd.dev/docs/storage#expiry)). The console's setup wizard writes this policy with your bucket's name for AWS, Wasabi and MinIO, and the matching key command for B2. For AWS it also gives you a CloudFormation template and a Terraform file. Both create the bucket with Object Lock and versioning on and public access blocked. The API returns the same text at `GET /api/v1/storage/bucket-setup?provider=aws&bucket=acme-safegrd`. A recovery machine needs only to read and list: ``` {"Version": "2012-10-17", "Statement": [ {"Effect": "Allow", "Action": ["s3:ListBucket", "s3:ListBucketVersions"], "Resource": "arn:aws:s3:::acme-safegrd"}, {"Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectVersion"], "Resource": "arn:aws:s3:::acme-safegrd/*"} ]} ``` [The recovery runbooks](https://safegrd.dev/docs/runbooks) use this key. ### A colder storage class Set `storage.storage_class` to choose the S3 storage class of each archive: `STANDARD` (the default), `STANDARD_IA` or `GLACIER_IR`. A colder class costs less to hold but charges for each GB read, and every Fire Drill reads one copy. It also has a minimum billed period (30 and 90 days on AWS), and you pay for that period even when the lock is shorter. Price your schedule and drills in the [Object Lock cost calculator](https://safegrd.dev/tools/object-lock-cost-calculator) before you change the class. The manifest beside each archive stays `STANDARD`, because every list and drill reads it. Incremental (`format: repo`) surfaces are not affected. Glacier Flexible Retrieval and Deep Archive are refused, because a drill could not restore from them. ## Cloudflare R2 R2 is Cloudflare's S3-compatible store, with no egress fees. It has no S3 Object Lock API. It refuses `PutObjectLockConfiguration`, so SafeGrd cannot lock a backup there or read a lock back. Set `worm_mode: "NONE"`, as for any [bucket without Object Lock](https://safegrd.dev/docs/storage#spaces). - Create an R2 bucket in the Cloudflare dashboard. - Under **Manage R2 API Tokens**, create an API token with **Object Read & Write** access for this bucket only. R2 tokens cannot be narrowed further, so this one can also delete. - Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in your environment. - Point the endpoint at your account URL: ``` storage: type: "s3" bucket: "acme-safegrd" region: "auto" endpoint: "https://.r2.cloudflarestorage.com" worm_mode: "NONE" ``` R2 has [bucket locks](https://developers.cloudflare.com/r2/buckets/bucket-locks/) of its own. A bucket lock keeps objects from being deleted or overwritten for a period you set on the bucket, from the dashboard or with `npx wrangler r2 bucket lock add `. If you want one, make it at least as long as `retention_days`. SafeGrd cannot see the rule, so the console still marks each snapshot **Not locked: the bucket has no Object Lock**. Anyone with permission on the Cloudflare account can remove the rule. Run `safegrd doctor` before the first backup to check the connection. ## MinIO (self-hosted WORM) MinIO has real S3 Object Lock when it runs in multi-drive or distributed erasure-code mode. - Create a bucket with Object Lock on, using `mc`: ``` mc mb --with-lock myminio/safegrd-backups ``` - Set a default retention: ``` mc retention set --default COMPLIANCE 30d myminio/safegrd-backups ``` - Save [the AWS backup policy above](https://safegrd.dev/docs/storage#aws), with your bucket's name, as `safegrd-backup.json`, and give the backup hosts a user that has only it: ``` mc admin policy create myminio safegrd-backup safegrd-backup.json mc admin user add myminio safegrd-backup CHOOSE-A-SECRET-KEY mc admin policy attach myminio safegrd-backup --user safegrd-backup ``` - Add the sink to `config.yaml`: ``` storage: type: "s3" bucket: "safegrd-backups" region: "us-east-1" endpoint: "http://minio.internal:9000" worm_mode: "COMPLIANCE" ``` `safegrd doctor` calls `GetBucketObjectLockConfiguration` to check the lock mode. ## Backblaze B2 B2 has Object Lock, which it calls File Lock. Turn it on when you create the bucket, because it cannot be added later. - Create a B2 bucket with **File Lock (Object Lock)** enabled. - Create an application key limited to that bucket, with only the capabilities a backup needs. Without `deleteFiles`, the key cannot delete or hide a file: ``` b2 key create --bucket acme-b2-safegrd safegrd-backup listBuckets,readBuckets,readBucketRetentions,listFiles,readFiles,writeFiles,writeFileRetentions ``` - Find your region endpoint (for example, `s3.us-east-005.backblazeb2.com`). - Configure the sink: ``` storage: type: "s3" bucket: "acme-b2-safegrd" region: "us-east-005" endpoint: "https://s3.us-east-005.backblazeb2.com" worm_mode: "COMPLIANCE" ``` ## Wasabi Wasabi has S3 Object Lock in compliance mode, and no egress fees. - Create a Wasabi bucket with **Enable Object Lock** turned on in **Compliance Mode**. - Under **Policies**, create a policy with [the AWS backup policy above](https://safegrd.dev/docs/storage#aws), with your bucket's name. Wasabi reads the same document. Create a user for the backup hosts, attach the policy, and make an access key. - Set the endpoint for your region: ``` storage: type: "s3" bucket: "acme-wasabi-safegrd" region: "eu-central-1" endpoint: "https://s3.eu-central-1.wasabisys.com" worm_mode: "COMPLIANCE" ``` ## DigitalOcean Spaces, and any bucket without Object Lock Spaces has no Object Lock. SafeGrd works there: snapshots are encrypted to your key, and Fire Drills test them as usual. But **nothing stops them from being deleted**. Anyone with a key that has delete permission on the Space can remove them. To use a Space, set `worm_mode: "NONE"` in the config: ``` storage: type: "s3" bucket: "acme-safegrd" region: "nyc3" endpoint: "https://nyc3.digitaloceanspaces.com" worm_mode: "NONE" ``` `safegrd init` writes the same from flags: ``` safegrd init --storage s3 --s3-bucket acme-safegrd --s3-region nyc3 --s3-endpoint https://nyc3.digitaloceanspaces.com --worm-mode NONE ``` The Space's key comes from `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in the host's environment, or from `access_key_id` and `secret_access_key` under `storage`. Without `worm_mode: "NONE"`, a backup to Spaces fails at startup, so an unlocked backup is never reported as locked. With it, the console marks each snapshot **Not locked: the bucket has no Object Lock**, and the CLI never prints an immutability date. `safegrd doctor` reports it as a warning: `worm_mode is NONE: snapshots in this bucket can be deleted by anyone with delete permission`. On a bucket that should lock and does not, `safegrd doctor` fails with `bucket ... cannot be confirmed to lock` and tells you what to change. Hand out as few keys to the Space as you can, write-only where possible. Consider also copying to a bucket that does lock. ## An SFTP server You can back up to a directory on any SSH server that offers SFTP, such as a NAS, a storage box or a machine of your own. Every surface type works there, in both formats (incremental and `format: tar`). Restores and Fire Drills read from it the same way as from a bucket. An SFTP server has no Object Lock, so set `worm_mode: "NONE"`. A config with any other value, or with none, is refused when it is read. Each snapshot records the date it is kept until, and the CLI never prints a lock date for it. ``` storage: type: "sftp" worm_mode: "NONE" retention_days: 30 sftp: host: "backup.example.com" port: 22 user: "safegrd" path: "backups" private_key: "file:/etc/safegrd/sftp_key" known_hosts: "/etc/safegrd/known_hosts" ``` `path` is the directory backups go in, either absolute or relative to the account's login directory. Sign in with `private_key` (a key without a passphrase) or `password`. Both take `env:VAR` or `file:/path`, so the secret stays out of the config. The CLI checks the server's host key before it sends anything. You can list the server in an OpenSSH `known_hosts` file: run `ssh-keyscan -p 22 backup.example.com >> /etc/safegrd/known_hosts`, then compare the fingerprints with the server's administrator. Or pin the key's fingerprint in place of `known_hosts`: ``` # on the server ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub # in the storage block, under sftp: host_key: "SHA256:..." ``` If the server's key does not match, or `known_hosts` does not list the server, the CLI refuses to connect, and the error shows the fingerprint the server presented. Give `known_hosts` as a full path, because the daemon runs as its own user, and that user's home is not yours. Give the account write access to `path` and nothing else. Anyone who can sign in as that account can delete the backups in it, so keep its key or password on the host that backs up. `safegrd init` does not write an SFTP block. Run it with `--storage local`, then replace the storage block with the one above. `safegrd doctor` signs in, lists the directory and reports that the server has no Object Lock. WAL archiving for point-in-time recovery does not use SFTP: it needs a bucket, hosted storage or a local directory. ## How WORM retention works `--retention-days` sets an S3 Object Lock retention period on each object as it is written. In **compliance** mode, the default, nobody can delete or overwrite the object until retention expires. That includes the account's root user and SafeGrd. Stolen administrator credentials cannot remove a locked backup either. ### You cannot shorten or undo a lock. Once a compliance lock is set, nobody can shorten or cancel it. With a long retention on hourly snapshots, you pay for storage you cannot delete until each date passes. Try a short `retention_days` on a small dataset first. `worm_mode: "GOVERNANCE"` is the softer mode. Anyone holding the bypass permission can shorten a lock. It protects against accidents, but an attacker with that permission can remove the backup. ### Keeping daily, weekly and monthly backups longer You can keep some backups longer than others (grandfather-father-son). Set the tiers on a surface or under `defaults`: ``` retention_days: 2 # every backup keep_daily: 14 # the first backup of each UTC day, for 14 days keep_weekly: 4 # the first backup of each ISO week, for 4 weeks keep_monthly: 12 # the first backup of each month, for 12 months ``` - The tier is decided when the backup is written, because an Object Lock period can never be shortened later. The first successful backup in a day, week or month takes that slot. A failed backup does not, so the next successful one takes it instead. The daemon logs which backup took a slot. On an hourly schedule, this keeps storage small: the hourly copies need only a short lock, and one backup a day carries the history. - If the daemon loses its state (a new host, a wiped state directory), the next backup takes the slots again. One backup may then be kept longer than planned, never shorter. - SafeGrd still never deletes a locked backup. The tiers decide how long each one is protected. A lifecycle rule on the bucket decides when it is deleted (see _Listing and expiry_ below). - A one-off `safegrd backup --database-url`, `--files` or `--email` applies the tiers under `defaults` too, or the ones you pass as `--keep-daily`, `--keep-weekly` and `--keep-monthly`. It records which day, week and month already have their copy, per source and bucket, in `one_off_slots.json` in the state directory (`~/.safegrd` unless `daemon.state_dir` says otherwise). A nightly cron line gets the same weekly and monthly copies a surface does. - If you pass `--retention-days` and no `--keep-*` flag, the backup is locked for exactly that many days and the config's tiers do not apply. The flag says what the lock is, and a lock cannot be shortened afterwards. - Under `worm_mode: "NONE"` nothing is locked, so the tiers protect nothing, and the daemon does not claim tier slots. ### Locking only the kept copies When you create a project, you choose which of its backups are locked. The organization's first project locks every backup. A project you create after that can lock only the copies it keeps: - The first backup of each day is locked when it is written, for the daily tier. On Scale, the first backup of each 6-hour slot is locked. - The weekly and monthly copies are chosen once the week or month has ended, from backups that a Fire Drill restored and Threat Shield did not flag. Each is locked for its tier. If none qualifies, the newest scheduled backup is kept anyway, marked not drilled, and you get one alert (`kept_undrilled`). - The backups in between are written unlocked and kept for the plan's retention. For one of these, `safegrd backup` prints "Kept until , not locked", and the console lists it as Recent. On hosted storage, they count at half against your included storage. - Nothing is deleted while a month is open. Once the month's repository is closed and the retention has passed, SafeGrd removes the unlocked objects that no kept copy references. In your own bucket, this needs the opt-in prune ([Keep one snapshot longer](https://safegrd.dev/docs/storage#keep) and `safegrd prune`). Without it, unlocked objects stay in the bucket. - While a Threat Shield anomaly is open, the objects of the last known good backup are locked 14 days ahead, and locked 14 days ahead again every day. - You cannot change a project's rule. To move a surface to the other rule, create a project and enroll its hosts there. - The rule applies to file and database surfaces, on hosted storage and in your own bucket. In your own bucket, the host applies it with its own key. Mailboxes and `format: tar` backups lock every copy, whatever the project's rule. ### Sinks with no Object Lock Use `worm_mode: "NONE"` for storage with no Object Lock, such as DigitalOcean Spaces. SafeGrd never switches to unlocked storage on its own. A backup to a bucket without Object Lock fails at startup unless this setting is there. With `NONE`, snapshots are written unlocked. The value is case-sensitive: `COMPLIANCE`, `GOVERNANCE` or `NONE`. ## Listing and expiry ``` # what is in the bucket, and until when it is locked safegrd list # sink reachability, lock configuration, key presence safegrd doctor ``` In your own bucket, SafeGrd deletes nothing until you turn pruning on. (Hosted storage prunes a snapshot once its retention ends.) When a lock expires, the snapshot may be deleted, but it stays until something prunes it. With pruning off, the bucket and its bill grow with every backup. A surface's panel in the console shows what it keeps now and what it will keep once its copies have rolled over. How fast it grows is in [the storage forecast](https://safegrd.dev/docs/storage#forecast). A local directory is never pruned. Once you no longer need its snapshot files, anyone who can write to the directory can delete them. To let expired snapshots go, set `storage.expire_after_lock: true` and the daemon prunes once a day. Or run `safegrd prune` yourself (with `--dry-run` first). Pruning deletes a snapshot only after its Object Lock ended more than a day ago. It judges that by the bucket's own clock and the lock the bucket itself reports, not by the host's clock. It keeps the newest three snapshots of each surface whatever their locks say (`--min-keep`), so a short retention never empties a surface. It also keeps the last known-good snapshot while Threat Shield reports an open anomaly. If SafeGrd's server is unreachable, nothing is pruned. Pruning needs `s3:DeleteObjectVersion`, `s3:GetObjectRetention` and `s3:GetObjectLegalHold` on the host's key. The wizard adds these when you turn on expiry. With them, the key can delete any snapshot whose lock has ended, and it still cannot touch a locked one. The bucket enforces the lock. `min_keep` and the anomaly hold are enforced by `safegrd` itself, so set `retention_days` to the furthest back you would need to restore. Neither the backup policy nor the recovery policy grants `s3:DeleteObject`, so a host's key cannot write a delete marker over a snapshot. Use pruning instead of a bucket lifecycle rule, because a lifecycle rule cannot tell the daily, weekly and monthly tiers apart. An [SFTP server](https://safegrd.dev/docs/storage#sftp) has no lock. There, `safegrd prune` and `expire_after_lock` delete a snapshot one day after the keep-until date it records. The day is judged by the server's clock: the time it stamps on a file it has just written. The newest `min_keep` snapshots and the anomaly hold apply there too. ### Keep one snapshot longer You can extend a snapshot's lock up to 13 months from today, before a migration or because an auditor asked for it. Extending is the one change Object Lock allows. Nobody can shorten the lock afterwards, SafeGrd included. On hosted storage, an owner or admin picks the date under the snapshot's _Details_ in the console. The kept snapshot counts against your hosted storage for as long as it is locked. In your own bucket, run `safegrd keep --snapshot --until 2027-01-31 --yes` on a host that writes to it. The host's key needs `s3:PutObjectRetention` and `s3:GetObjectRetention`. The console shows the new date once the host reports it. The backup file, its metadata and its recovery document are extended together. An incremental snapshot shares its chunks with the rest of its month, so it is kept as long as that month and cannot be kept on its own. ### When a project changes buckets If you move a project to another bucket, or to or from hosted storage, or delete an organization, the backups already written stay where they are. SafeGrd records what was left behind. The key access log gets a _sink_released_ entry naming every snapshot SafeGrd recorded in the old location, and the console tells you the same when you make the change. In your own bucket, your lifecycle rule or `safegrd prune` removes them once their locks end. Hosted storage removes each one when its lock ends. ## A second bucket You can have every backup written to up to two more S3 buckets, in another account, at another provider or in another region. Each bucket has its own access key, lock and retention. Every copy is encrypted to the same Age key as the first, so the key that restores one restores them all. ### Add a bucket in the console On Growth and Scale: - Create the bucket at your provider. Turn on Object Lock if you want its copies locked. Make an access key that can write to it. - Open **Projects**, click **Storage** on the project, and go to **More buckets**. - Click **Add bucket**. Give it a name, such as `offsite`, and fill in the bucket, endpoint, region and keys. - Choose its lock and how many days to keep each backup. Click **Test**, then **Save**. Every host in the project writes to the bucket from its next backup. SafeGrd holds the bucket's key, encrypted, and gives it only to your enrolled hosts. ### Or list it in the host's config This works on every plan. Add the buckets under `copies:` in the host's config: ``` copies: - name: offsite type: s3 bucket: company-backups-offsite endpoint: https://s3.us-west-004.backblazeb2.com region: us-west-004 worm_mode: COMPLIANCE retention_days: 90 access_key_id: env:B2_KEY_ID secret_access_key: env:B2_KEY ``` Every surface on the host writes to every bucket in the list. To choose for one surface, set `copies: [offsite]` on it, or `copies: []` for none. If the project has buckets set in the console, those replace this list, and the host says so on each run. Every key is listed in [the config reference](https://safegrd.dev/docs/config#copies). ### See what each bucket holds Each snapshot in the surface's **Snapshots** tab says how many copies it has, such as "2 of 2 copies". If a bucket is behind, the surface shows **Copy behind**. On the host, `safegrd copies status` lists what each bucket lacks. ### How a backup reaches each bucket - A full backup is one archive file. It is written to every bucket at once, from the same encrypted stream. Nothing is read back out of your own bucket. - An incremental backup is written to every bucket during the run, object by object, as the host reads it. If a bucket missed part of the month's backups (it was down, or you added it mid-month), it is filled after the run from your own bucket. Your provider bills that read as egress. Databases back up incrementally unless the surface was added with full backups. - Each bucket locks a backup with its own `worm_mode` and `retention_days`. An incremental run stays locked for the bucket's retention from the end of its month. A bucket set to `NONE` is shown as not locked. ### When a bucket is down If one bucket is down, the backup still succeeds on your other buckets. The host prints `Warning: copy to offsite failed`, and the surface shows **Copy behind**. The daemon tries every half hour, and once the bucket answers, it writes what the bucket lacks. After a day behind, you get a `copy_behind` alert, and a `copy_caught_up` alert when the bucket holds every backup again. To catch up now, run: ``` safegrd copies sync ``` ### Restore from a copy A copy holds the same files as your own bucket, so you restore from it the same way: ``` safegrd restore --snapshot snap-1b094c0af8 --from-bucket offsite --target "$DATABASE_URL" ``` `verify`, `list` and `find` take `--from-bucket` too. Without SafeGrd, you restore from a copy the same way as from your own bucket ([runbooks](https://safegrd.dev/docs/runbooks#without-safegrd)). To test that each copy restores, set `drill.source: rotate` on the surface. Each Fire Drill then reads the next bucket, and its certificate names the bucket it read. ### Limits - A surface that SafeGrd backs up without a host of yours is not copied to more buckets yet. - You cannot move or remove a bucket in the console while it holds locked copies. Add a new bucket under another name instead. ## A recovery document beside every snapshot Every snapshot is written with two files beside it: `SNAPSHOT.meta.json` (the manifest) and `SNAPSHOT.RECOVERY.md`, a plain-text recovery page. The page says: - where the snapshot is (bucket, endpoint, prefix) - what a restore needs: the server version, and the extensions and roles to have ready - the commands that restore it, with `safegrd restore` and, for PostgreSQL, with `psql` alone It is written for someone who has the bucket and the key and nothing else. So it is not encrypted, and it holds no secret: no credential, no key and no connection string. For a full backup, the two files sit next to its archive. For an incremental snapshot, they sit in the `snapshots/` folder of its month's repository, `repo///snapshots/`. On hosted storage, download them from the snapshot's details in the console. Set `recovery_doc_sealed: true` on a surface to seal the recovery page with the surface's recipient instead, as `SNAPSHOT.RECOVERY.md.age`. Both forms are locked like the manifest. If the recovery document cannot be written, the CLI says so on stderr, and the backup still succeeds. ## Someone deleted my backups You can bring them back with `safegrd undelete`. In a versioned bucket with Object Lock on, an ordinary delete cannot remove a locked object. S3 puts a _delete marker_ on top of it instead. The marker hides the snapshot from normal listings, and the data stays where it was. ``` # make one snapshot visible again safegrd undelete --snapshot snap-1b094c0af8 # clear every delete marker in the sink safegrd undelete --all ``` `safegrd undelete --all` rolls a versioned bucket back to before the deletion. Every snapshot and sidecar is visible again once its delete marker is gone. Nothing is re-uploaded, so the rollback takes seconds however much the bucket holds. The command removes only delete markers, each addressed by its version id. It cannot remove a version that holds data, so it is safe to run twice. If it finds nothing to remove, it tells you why: either the snapshot is there and was never hidden, or the id is not in that bucket at all. In the second case it names the bucket, prefix and node it searched, and exits non-zero. Removing a delete marker needs `s3:DeleteObjectVersion`. A host's key has that only when expiry is on, so run the command with an administrator's key. Afterwards, deny `s3:DeleteObject` to the credential that made the deletion. --- Source: https://safegrd.dev/docs/daemon # Daemon The daemon runs the backups in your config file on schedule. On each pass it reads the config, finds the surfaces that are due, locks each one and backs it up. It also reports the host's health to SafeGrd, so you are alerted if the host stops responding. ## Run it ``` # resident daemon safegrd daemon run # one reconciliation pass and exit, for cron or a Kubernetes Job safegrd daemon run --once # what every surface is doing, and when each last succeeded safegrd daemon status ``` The first pass backs up every surface that is due. A new surface is due at once: ## Install it as a service ``` # systemd on Linux, launchd on macOS safegrd daemon install # as a user agent rather than a system service safegrd daemon install --user # print the unit and install nothing, to review or commit it safegrd daemon install --print ``` The service runs the same binary with the config you installed it from (the one `--config` names, or `~/.safegrd/config.yaml`). It backs up exactly what `safegrd daemon run` backs up in your shell. The daemon keeps its state and locks in `~/.safegrd` of the user who ran `install`. For a system service installed with `sudo`, that is `/root/.safegrd`. A system service sees the filesystem as read-only, except for that directory and a local sink. `install` creates both if they do not exist yet. A user service stops when you log out unless you run `loginctl enable-linger $USER` once. To restart the service, run `safegrd daemon restart` (add `--user` for a user service). `safegrd daemon uninstall` removes the unit file and prints the command that stops a service that is still running. Install it as a system service or as a user service, not both. Two daemons writing to the same sink as different users leave directories that one of them cannot write to. ### Config changes and upgrades The daemon reads the config again at the start of each pass if the file changed. A surface you add by hand or with `safegrd claim` ([details](https://safegrd.dev/docs/surfaces#adding)) is backed up without a restart. If the changed config does not load, the daemon logs it and keeps the config it had. Changes to `metrics_addr`, `interval`, `updates` and SQLite followers take effect at the next `safegrd daemon restart`. So does an upgrade you install by hand. A running daemon keeps running the binary it started with and logs once that the file was replaced, so run `safegrd daemon restart` after you upgrade. An update from the console needs no restart ([below](https://safegrd.dev/docs/daemon#updates)). ## Schedules Each surface's `schedule` sets how often it backs up. The daemon's `interval` is only how often it wakes up to _look_ for work. The daemon adds about 10% jitter, so a fleet of hosts does not hit the same bucket on the same minute. | Form | Meaning | | --- | --- | | @hourly, @daily, @weekly | The usual shorthands. | | 6h, 90m | A plain duration between runs. | | 2d | A whole number of days. | | at 02:00 | Once a day, at that time. | | 0 3 * * 1-5 | A cron line: minute, hour, day of month, month, day of week. This one is 03:00 on weekdays. | A schedule cannot run more often than once an hour. `safegrd config validate` refuses a shorter one, and a cron line whose two closest runs are less than an hour apart. Under compliance-mode Object Lock, each snapshot stays locked until its lock date, so frequent backups add up: a 30-minute schedule with 30-day retention would keep 1,440 snapshots. Times of day and cron lines are in UTC. Set `clock: local` on a surface to read them, and its blackout windows, in the host's time zone. The surface backs up at the first such time after its last backup. For a cron line, Overdue uses the longest gap between two runs: three days for `0 3 * * 1-5`, Friday to Monday. Owners and admins can also set a surface's schedule and retention in the console: open the surface's Details under Surfaces and click Edit. The daemon picks up the new settings at its next check-in and uses them instead of the config's until someone clears them in the console. The config file on the host does not change. The daemon logs this when the settings change and once each time it starts, and `safegrd daemon status` marks such a schedule `(console)`. Until the daemon reports that it runs a setting, the console shows it as waiting. Use `blackout` to list times of day when the daemon starts no scheduled backup for a surface, for example `blackout: ["22:00-02:00"]`. The times are in UTC, or on the surface's `clock`. A backup that falls due inside a window waits until the window ends, and the daemon logs that it is waiting and until when. A backup that is already running when a window opens runs to the end. _Back up now_ from the console is not held back. Overdue waits longer by the total length of the surface's windows. `safegrd config validate` checks each window. To pause a surface's schedule, click _Edit_ on its row in the console, then _Pause the schedule_. Give a reason and, if you want, a time to resume. While the surface is paused, no scheduled backup or drill runs and it is not reported Overdue. _Back up now_ and _Drill now_ still run. WAL archiving goes on, and its point-in-time drill waits with the others. The daemon logs the pause at its next check-in and keeps it while SafeGrd's server cannot be reached. Click _Resume_ on the row to end the pause. The backup that is due then runs at the next check-in, and Overdue counts from the resume. Pausing works the same way for a surface SafeGrd backs up itself. The activity log records who paused it and why. The 30-day history shows the paused days as paused, so they do not count as missed, and the worst gap leaves the paused time out ([recovery figures](https://safegrd.dev/docs/verify#recovery)). Set `skip_if_unchanged: true` on an incremental surface to skip writing a backup when nothing changed since the last one: no new data, and no file or table added, changed or removed. The run still reads the whole surface to find that out. The daemon logs the skip and the schedule starts again from that run. The console shows the surface as unchanged since its last backup instead of Overdue. Drills keep restoring the last backup. This works on your own bucket and on local storage. On hosted storage every run writes a backup, and the daemon says so when it starts. On every pass, the daemon works out each surface's next run from the host's clock. If the clock goes back (a bad NTP step, or a VM restored from an image), a surface whose last backup is now in the future backs up at once instead of waiting for the clock to catch up. If the clock moves by more than `interval` between two passes, the daemon logs how far and in which direction. A machine that was suspended shows up as a step forward. ## The daemon block | Key | What it does | | --- | --- | | interval | How often the daemon looks for work. Default `5m`. | | state_dir | Where the daemon keeps state, locks and logs. Do not put it on a network filesystem. The lock on a backup in progress needs `flock` to work. | | metrics_addr | The address to serve Prometheus metrics on, at `/metrics`. See [Metrics and pings](https://safegrd.dev/docs/daemon#metrics). | | ping_url | A check URL the daemon pings after every pass. See [Metrics and pings](https://safegrd.dev/docs/daemon#metrics). | The daemon backs up one surface at a time. It tries a failed backup 3 times in all, 5 minutes apart, then waits for the surface's next scheduled run. It logs plain text to stdout and stderr, which systemd and launchd keep. ## Credentials the daemon needs A daemon cannot ask you for a password. Instead, each surface's `credential` block says where to read its secret: - `from: env` reads the environment variable named in `name`. - `from: command` runs `run` (your own secret manager) and reads the value from its output. - `from: file` reads the file at `path`. - `from: safegrd`: SafeGrd keeps the secret encrypted and gives it only to this host when the surface backs up. The first three never send the secret to SafeGrd. - For a database, the secret is the whole connection URL. For a mailbox, it is the password. A mailbox that names no credential uses `SAFEGRD_EMAIL_PASSWORD`. - If the block names something the daemon cannot read (an unset variable, a failing command or a missing file), that surface fails. The daemon does not fall back to anything else. - A `from: command` runs under `/bin/sh` as the daemon's user, with a 30-second limit. If it fails, that surface fails with the command's error output, and the other surfaces carry on. It does not fall back to an environment variable. - A service does not inherit your login shell. Run `safegrd doctor`, which resolves every surface's credential the way the daemon will. Then run it again as the service's user. A secret manager you are signed in to may not be signed in for that user. - The config file must be readable and writable only by its owner (0600), so nobody else can change these commands or read the credentials. ### An environment variable for the service The service that `daemon install` writes does not see variables you export in a shell. Add them with a systemd drop-in, which stays in place when the unit is reinstalled: ``` sudo systemctl edit safegrd # in the editor: [Service] Environment=APP_DATABASE_URL=postgres://backup:PASSWORD@db.internal:5432/app sudo systemctl restart safegrd ``` Any user on the host can read an `Environment=` line with `systemctl show`. On a shared host, use `from: file` with a file only the daemon's user can read (0600), or `from: safegrd`. ## Before and after a backup You can run your own command before and after each backup of a surface, for example to flush a cache, or to pause a writer and resume it afterwards: ``` pre_backup: "redis-cli BGSAVE && sleep 5" post_backup: "systemctl reload app" ``` - If `pre_backup` fails, the daemon skips that backup and counts a failure for the surface. The hook's output goes to the daemon's log. Set `pre_backup_on_error` to change this. `retry` runs the hook again after 10 seconds and then after 20 before skipping the backup. `continue` takes the backup anyway and logs a warning. The default is `skip`. - `post_backup` runs after every attempt, so whatever `pre_backup` paused is resumed even if the backup failed. It gets `SAFEGRD_BACKUP_STATUS` (`success` or `failed`) and `SAFEGRD_SNAPSHOT_ID`. - `post_drill` runs after each Fire Drill the daemon runs, for example to run a smoke test or warm a cache. It gets `SAFEGRD_DRILL_STATUS` (`passed`, `shallow`, `failed` or `blocked`) and `SAFEGRD_SNAPSHOT_ID`. If the sandbox still holds the restored data, it also gets `SAFEGRD_SANDBOX_URL`. That is a throwaway PostgreSQL on the host, or a sandbox kept after a failed drill. If the hook fails, the daemon logs it and nothing else changes. - All three hooks run under `/bin/sh` as the daemon's user, with a 10-minute limit each. They live in the config file, which is another reason it must be readable only by its owner. ## What the console sees Each surface in the config shows up in the console on its own, under the host you enrolled, from the first time the daemon sees it. The host's token covers all of its surfaces, so there is one credential per host, and rotating it covers them all. A new surface counts against your plan like any other. On every tick, the daemon checks in for each surface. That lets the console tell _the daemon has stopped_ apart from _this one surface is failing_: - _Silent_: the daemon has missed three check-ins (and at least 15 minutes). Nothing is backing up that host. - _Overdue_: a surface has gone twice its schedule, plus its [blackout windows](https://safegrd.dev/docs/daemon#blackout), without a backup. This also applies to hosts that run `daemon run --once` from cron. Those do not count as daemons unless each pass names its schedule with `--interval`, for example `daemon run --once --interval 5m` from a cron line that runs every 5 minutes. - _Failing_: the last attempt failed, with the error the daemon saw. - _Unreachable_: on three ticks in a row, the daemon could not open the surface’s database or reach its sink. The row shows since when, and the error. ### Checks between backups If a database goes away at 09:00, the console shows it a few ticks later, without waiting for the next scheduled backup. On each tick, before it checks in, the daemon opens the database of each PostgreSQL, MySQL and MongoDB surface and runs `SELECT 1` (or the engine’s ping). It also checks that the sink answers, with a `HeadBucket` on your own bucket or a look at the local directory. Each check gives up after 10 seconds. The checks read nothing else and write nothing, with one exception. For a surface that names `erasures:`, the daemon moves the waiting rows of `safegrd.erasures` into the surface’s storage ([the erasure record](https://safegrd.dev/docs/surfaces/postgres#erasures)). If that move fails, it counts as a sink that does not answer. These surfaces are checked only when they back up: - A surface whose credential SafeGrd holds, because the daemon fetches that credential for a backup or a drill and for nothing else. - A surface whose credential comes from `credential.from: command`, which would otherwise run that command every tick. The daemon does not check hosted storage, or a bucket whose access key SafeGrd holds. A surface with nothing to check is shown as its last backup left it. **Back up now**, under a surface’s **Details** in the console, tells the daemon to back the surface up at its next check-in. If the previous backup was less than an hour ago, the daemon skips the request, so locked storage does not fill with redundant snapshots that cannot be deleted. ## Alerts On any plan, add a Slack, Discord or plain webhook URL under _Settings → Alerts_ in the console. Paid plans also email the organization's owners and admins. You get an alert for a failed backup, a failed Fire Drill, a drill that cannot run because the key is missing, a silent host, and an overdue or unreachable surface. You get another when the backup, host or surface recovers. Each is sent once, not on every check. [Alerts](https://safegrd.dev/docs/alerts) lists every event and what each message contains. ## Metrics and pings Set `daemon.metrics_addr: "127.0.0.1:9847"` and the resident daemon serves Prometheus metrics at `http://127.0.0.1:9847/metrics`. Each scrape reads the state the daemon wrote after its last backup. Every series except the first two has `surface` and `type` labels: | Metric | What it is | | --- | --- | | `safegrd_daemon_info` | 1, with the daemon's `version` | | `safegrd_daemon_state_updated_timestamp_seconds` | When the daemon last wrote its state | | `safegrd_surface_last_attempt_timestamp_seconds` | When the last backup started | | `safegrd_surface_last_success_timestamp_seconds` | When the last backup succeeded | | `safegrd_surface_consecutive_failures` | Failed backups since the last success | | `safegrd_surface_next_due_timestamp_seconds` | When the next scheduled backup is due | | `safegrd_surface_last_backup_duration_seconds` | How long the last successful backup took | | `safegrd_surface_last_backup_size_bytes` | What it stored, encrypted | | `safegrd_surface_last_drill_timestamp_seconds` | When the last Fire Drill ran | | `safegrd_surface_drill_passed` | 1 when the last Fire Drill passed, else 0 | A timestamp for something that has never happened is left out instead of written as 0. To alert when the last successful backup is more than a day old: ``` time() - safegrd_surface_last_success_timestamp_seconds > 86400 ``` Bind it to `127.0.0.1` unless the scraper runs on another machine, because the metrics include your surface names. To use a check service such as Healthchecks.io or Cronitor, give a surface a `ping_url`. The daemon posts to `URL/start` before each backup and to `URL` after it. If the backup failed, or SafeGrd did not record it, the daemon posts to `URL/fail` instead. Each post carries one line about what happened as its body. The daemon pings `daemon.ping_url` after every pass, so the check alerts you when the daemon stops running. Both settings take `env:VAR` or `file:/path`, because anyone who has a check URL can report to that check. If a ping fails, the daemon prints a warning to stderr and the backup's result stands. ``` daemon: metrics_addr: "127.0.0.1:9847" ping_url: "env:HEALTHCHECK_DAEMON" surfaces: - id: "orders-db" type: "postgres" ping_url: "https://hc-ping.com/your-check-uuid" ``` ## Fire Drills on an unattended host When a surface is due for a drill on your plan's schedule, SafeGrd's server tells the daemon. The daemon restores the latest snapshot, checks it against the digests recorded when it was taken, and reports the result to your attestation record. A drill needs the private key, so the daemon runs one only on a host that already has it (through `key_path`, `SAFEGRD_PRIVATE_KEY`, or a SafeGrd-managed key). Every surface in an organization is sealed to that one key, and the daemon never copies it to a host. A host without the key reports _Drill blocked: no key_, and you are alerted. You can then run `safegrd verify` wherever you keep the key. For a PostgreSQL surface, you can give the drill a scratch database. The drill then restores the rows into it and counts them there. Without one, the snapshot is replayed in memory, unless the host can run a throwaway cluster (below). ``` - id: "app" type: "postgres" credential: from: env name: "APP_DATABASE_URL" drill: sandbox_url: "env:APP_DRILL_DATABASE_URL" ``` Create the sandbox once (`createdb app_drill`) and give the daemon a role that owns it. Use it for nothing else, because a drill restores into it. The daemon refuses a sandbox that holds any table, or one that is the database the surface backs up, and it empties the sandbox after every drill. If your plan includes sandbox drills, a PostgreSQL surface with no `drill.sandbox_url` gets a scratch database on the host. When the host has the PostgreSQL server installed (`initdb` and `postgres`, of the source database's major version or newer), the daemon creates a throwaway cluster under its state directory and restores the snapshot into it. It deletes the cluster when the drill ends. You do not need Docker. - The cluster listens on a Unix socket only, in a directory only the daemon's user can open. - It uses under 200 MB of memory, whatever the size of the database. Restored rows go to disk. - It needs about as much disk as the database. The daemon checks first, and does not start a cluster that would leave less than 5% of the filesystem free (1 GiB on a disk over 20 GiB). - PostgreSQL does not run as root, so a daemon running as root runs the cluster as the user who owns its state directory. ### Look at a failed drill's sandbox Set `drill.keep_failed_sandbox: true` to keep the sandbox as it was for 24 hours when a drill restores the snapshot and then fails a check. The daemon prints where the sandbox is, without the password: ``` Warning: Surface app: sandbox kept at postgres://drill:xxxxx@db.internal/app_drill until 2026-10-06T09:25:25Z; the first drill after that empties it. ``` A `sandbox_url` database is left alone for those 24 hours. The next drill is held back, and the first drill after the 24 hours empties it. A throwaway cluster keeps running on its socket until the daemon removes it. Everything stays on the host. If the restore itself fails, it is rolled back and there is nothing to keep. The error is in the daemon's log and in the console. ### Install a PostgreSQL server for sandbox drills On Debian and Ubuntu, install the server with `apt install postgresql-16`. The `postgresql-client` package has no server. The server package also starts a server of its own, which the daemon does not use. Stop it with `systemctl disable --now postgresql`. On macOS, run `brew install postgresql@16`. If the daemon cannot start a cluster (no server, a server older than the database, too little disk, or an extension the snapshot uses that the server lacks), the drill runs in memory. The daemon's log says what to install. When a drill is not the kind your plan includes, the console says why under the surface: it ran in memory because there was no sandbox, or it did not run because the host lacks the disk. A drill that does not run is not a failed backup. The daemon tries again at the next drill, and you are alerted once. ## Disk the daemon uses on the host Database and file backups stream to storage with no copy on the host. Four things do write to the host's disk. Before each one, the daemon checks that it would leave 5% of the filesystem free for everything else on it (at most 1 GiB): - A drill of an incremental file surface restores the whole tree under the state directory, then deletes it. If there is no room, the drill does not run. - A local sandbox drill needs about as much as the database, as above. - A SQLite backup copies the database to `TMPDIR` first. If there is no room, the backup fails and says so. Point `TMPDIR` at a larger disk. - A local sink (`storage.type: local`) holds every backup. A backup stops before the free space would drop below that limit, and the error says what to free. ## Updates from the console When a new release is out, the daemon downloads and checks it, and the host's row under Hosts shows _ready to install_. An owner or admin clicks **Update**, and the daemon installs the release at its next check-in (every 5 minutes by default), between backups. Before it switches, it checks that: - the archive matches the SHA-256 that the release's `checksums.txt` lists - the release is newer than the one you run - the new binary loads your config and the daemon's state with the flags your service passes SafeGrd's server only tells the host which release to install. The host downloads it from [the release page](https://github.com/safegrd/cli/releases) (or from `daemon.updates_url`), so SafeGrd's server never sends you a binary. The host keeps the release it ran before. If the new one does not check in within two passes, or starts three times without checking in, the host switches back to the earlier one on its own, and the console says why. To go back yourself, click **Go back**. The earlier release is still on the host, so nothing is downloaded. Console updates work on hosts set up with `install.sh`. It keeps each release in `/usr/local/lib/safegrd/versions` (`~/.local/lib/safegrd` without sudo) and points `/usr/local/bin/safegrd` at the one in use. On a host installed before this existed, run the installer once. If you installed with Homebrew, the container image or `go install`, update the usual way, and the console tells you which command to run. To turn console updates off on a host, set `daemon.updates: off`. ## When SafeGrd's server cannot be reached The daemon keeps running scheduled backups and writing them to your storage. Each snapshot is in your bucket and restores, and `safegrd list` shows it. The daemon prints that it could not send the backup's record, and keeps the record. On the first tick the server answers, the daemon sends it and the console lists the backup. `safegrd daemon status` shows how many records are waiting. If the server refuses a report (because the token expired, for example), the daemon does not send it again. It is printed to stderr and shown as `not recorded`. A backup you run by hand with `safegrd backup` keeps no state, so its record is not sent again. With `--surface`, the backup goes through the daemon's state, and its record is. --- Source: https://safegrd.dev/docs/hosted # Hosted backups and drills If your database runs on Supabase, Neon or RDS and you have no server beside it, SafeGrd can back it up for you. If a surface has no host to drill it, for example one you back up from a GitHub Actions workflow, SafeGrd can drill it too. For each job, SafeGrd starts a machine, gives it the one secret that job needs, and destroys the machine when the job ends. ## Back up a database with nothing to install - Under **Surfaces**, choose **Add surface**, then **Back up on SafeGrd**. - Name it, and pick what it is: PostgreSQL, MySQL or MariaDB, MongoDB, or a bucket. - Paste the connection string. **Test connection** opens it from SafeGrd's server and says what failed when it cannot. - Pick how often to back up and how long to keep each backup, then **Add and back up now**. The first backup starts within a few minutes. If the password is wrong, you find out then instead of the next morning. The first drill runs on your plan's schedule. The database has to accept connections from the internet. A [read-only role](https://safegrd.dev/docs/surfaces/postgres#role) is enough. If your database only accepts listed IP addresses, the console shows SafeGrd's addresses beside _Back up on SafeGrd_ (when SafeGrd's machines connect from fixed ones). ### Supabase Click _Sign in to Supabase_ and pick your project from the list. The session pooler's address is filled in for you. From then on, each backup signs in through Supabase with a read-only role whose password expires within minutes. You can type the project's database password instead, if you have it (Supabase does not show it again). To skip the sign-in, copy the Session pooler string (port 5432) from _Connect_ in Supabase's dashboard and add `?sslmode=require`. The transaction pooler on port 6543 cannot take a backup. A project's Storage files are a second surface. You can add them two ways: - With the sign-in: tick _Also back up its Storage files_ beside _Sign in to Supabase_, where the console offers it. You type no S3 key. For each backup, SafeGrd reads the project's keys through the sign-in and hands them only to the machine it starts for that backup. They are never stored, and each use shows in the key access log. - By hand: add a bucket surface with the project's S3 endpoint and an S3 key from _Storage, Settings_. Use this if the project has its legacy JWT keys turned off, because Supabase's S3 endpoint accepts those keys as the temporary key. The console tells you when this applies. Each bucket is backed up as one archive of its objects and drilled on its own ([details](https://safegrd.dev/docs/surfaces/files#bucket-hosted)). Both surfaces appear together under _Supabase project_ in the surfaces list, each with its own Fire Drills and certificate. To restore, bring back the database rows first, then run `safegrd restore --target s3://*` with the project's S3 key. Each file goes back to the bucket it came from. ### Each backup uploads what changed A PostgreSQL backup reads every table and uploads only the rows that changed since the last one. ### What it costs at your provider Every backup reads the whole database, and your provider may bill that as egress. Before you add the surface, click **Price this schedule** in the form. It measures the database and prices the schedule at its provider. From a terminal, [`safegrd estimate`](https://safegrd.dev/docs/cli#estimate) does the same. ## Drill a surface on SafeGrd Turn on **Drill on SafeGrd** in a surface's panel. Each time a drill is due, SafeGrd starts a machine, restores the newest snapshot, checks it the same way a host would, signs the record and destroys the machine. The Fire Drills tab marks these drills as run on SafeGrd. Surfaces that SafeGrd backs up are drilled this way from the start. The surface's backups have to be in hosted storage, with a [SafeGrd-managed key](https://safegrd.dev/docs/security#key), because the machine needs to decrypt the snapshot. If the option is missing, the surface's panel tells you why. PostgreSQL, MySQL, MariaDB, MongoDB and SQLite snapshots are restored into a throwaway database of the same engine. Files and mailboxes are read back in memory. If a PostgreSQL snapshot needs an extension the plain image does not have, such as pgvector or a Supabase extension, the drill runs on Supabase's own image, which has them. [Drill checks](https://safegrd.dev/docs/verify#checks) you wrote as queries run on SafeGrd's machine too. Checks written as commands run only on your own host. ## A run on SafeGrd outside the schedule On a surface's Details, _Back up now on SafeGrd_ and _Drill now on SafeGrd_ start a machine straight away, even for a surface a host normally runs. Use them when the host is down, or when an audit is coming and the plan's next drill is days away. Agents can do the same with the MCP tools `request_backup` and `request_drill` and `on: safegrd`. ### What you need The host needs a SafeGrd-managed key. A drill also needs the newest backup to be in hosted storage. A backup needs SafeGrd to hold the surface's connection string (_Credential_ on the surface) and the project to use hosted storage. The backup is sealed to the host's key, so the host can restore it. If something is missing, the button says what. ### What it costs Each plan includes a number of these runs per rolling 30 days (see the table below). Backups and drills share the count across the whole organization. - A drill that the plan's schedule already allows is free. A drill before it is due costs one run. - A backup on SafeGrd of a surface that a host runs always costs one run. - Every run also counts its machine minutes, like any other job. Once you have used them all, the button tells you when the oldest run drops out of the 30 days. ### What it records A drill started this way is a normal Fire Drill. It is signed into the record and shown on the certificate, and the next scheduled drill is due one interval after it. A backup appears under the same surface as the host's own backups. On an incremental surface, that run uploads the whole database, because the machine does not have the host's cached state. ## Which machine gets which secret Each machine gets only the secrets its job needs, for that one surface, and only while the job runs. | Job | Given | Never given | | --- | --- | --- | | Backup | The connection string or bucket key, and the public key it encrypts to. | The key that decrypts backups. | | Fire Drill | The key for the snapshot it restores. | The connection string. | SafeGrd writes each release to the _Key access record_ before it happens, with the job and the time. You can read it under **Projects**, **Storage**. For what SafeGrd can hold and the two ways your key can be managed, see [Security and key custody](https://safegrd.dev/docs/security). ## Minutes on SafeGrd Each plan includes machine time per rolling 30 days, and a size limit for snapshots drilled in a sandbox: | Plan | Runs on SafeGrd | Largest snapshot | At once | On-demand runs | Surfaces SafeGrd backs up | | --- | --- | --- | --- | --- | --- | | Free | 20 minutes a month | In memory, within the plan’s storage | 1 | None | None | | Starter | 800 minutes a month | 5 GB | 1 | 10 runs a month | 3 | | Growth | 2,500 minutes a month | 8 GB | 2 | 30 runs a month | 10 | | Scale | 8,000 minutes a month | 25 GB | 4 | 100 runs a month | 30 | A job counts the time its machine ran, from when SafeGrd asked for the machine until the job ended. If the job moved a lot of data, it counts 10 minutes per GB instead, whichever is more. For example, a drill of a 2 GB snapshot that runs for 6 minutes counts 20 minutes. A backup of a 300 MB database that runs for 3 minutes counts 3. Fractions of a minute count, and there is no minimum. - While a job runs, the console counts it at the most it can use: 10 minutes per GB of its snapshot. A first backup has no known size yet, so it counts as the largest snapshot your plan drills. When the job ends, the total drops to what it used. - A stalled job is stopped at its deadline: 5 minutes, plus 5 per GB of the snapshot. - Once you use up the plan's minutes, the next run waits until enough of the 30 days has rolled off. The console shows when. - A snapshot over the sandbox limit is refused before anything is downloaded. Drill it on one of your own hosts with `drill.sandbox_url` instead. - Drills on your own hosts are never counted. --- Source: https://safegrd.dev/docs/verify # Restore and Fire Drills You can restore any backup from the command line or the console. On a schedule, SafeGrd also restores each surface's newest backup, checks what came back and signs a dated record of the result. ## Restore a backup ``` # into a PostgreSQL target safegrd restore --snapshot snap-1b094c0af8 --target "$DATABASE_URL" # files or mail, onto disk safegrd restore --snapshot snap-94dec39a5e --target-dir ./recovered ``` `restore` downloads the encrypted backup from storage, decrypts it on this machine with your private key, decompresses it and writes it to the target. It reads the key from `key_path` or `SAFEGRD_PRIVATE_KEY`. On a host other than the one that took the backup, pass `--key-path` or `--private-key`. ## Restore from the console On a paid plan, you can restore any completed snapshot from the console. Your host runs the restore, and the console shows how it went. - Create a database account that can write to the target. The backup's own role is read-only, so it cannot restore. - Click **Restore** on the backup you want. The button is on the **Snapshots** page, in the **Snapshots** tab of the backup's surface, and at the top of the backup's detail. - Choose what to restore: the whole backup, into an empty database - chosen schemas (PostgreSQL only) - chosen tables, from any PostgreSQL backup or from an incremental backup of MySQL, MariaDB, SQLite or MongoDB (one collection) - the rows of one table that a SQL condition matches (PostgreSQL only). They go back by primary key into the table as it is in the target. A files surface restores into an empty directory on the host, or into a bucket. You can also start from the Tables tab of a database surface. **Restore table** on each table opens the dialog with that table picked, and so does Restore rows on a PostgreSQL surface. - Type the target's URL with the password of the account from step 1. - Read the summary, type the surface's name, and press **Restore**. The restore appears in the surface's **Restores** tab. It waits there until your host checks in, then shows how many tables and rows came back. Only owners and admins see the **Restore** button. On Free, the console shows the `safegrd restore` command to run on your host instead. The console does not restore WordPress sites or Laravel apps. Restore a WordPress site from wp-admin and a Laravel app on its host. ### Restore to a moment For a surface with point-in-time recovery, open its **Point in time** tab and pick a moment inside the window. The console writes the `safegrd wal restore` command for that moment. Run it on a host that has the surface's config and key. Each database's page has the steps: [PostgreSQL](https://safegrd.dev/docs/surfaces/postgres#pitr), [MySQL and MariaDB](https://safegrd.dev/docs/surfaces/mysql#pitr), [MongoDB](https://safegrd.dev/docs/surfaces/mongodb#pitr) and [SQLite](https://safegrd.dev/docs/surfaces/sqlite#pitr). ### Where the restore runs - If your host runs the daemon, the host starts the restore at its next check-in. - For a surface SafeGrd backs up itself, a machine SafeGrd starts for the restore runs it. The surface needs a SafeGrd-managed key, and the backup has to be in hosted storage. - Otherwise, the console gives you `safegrd restore --request rst_...`. Run it on any host in your organization that has the key. It reads what you chose in the console and reports back. ### Your target and its password - Nothing in the target is dropped or overwritten. If a table already exists and has rows, the restore stops and leaves the target as it was. - SafeGrd keeps the URL you typed sealed and gives it once to whatever runs the restore. The key access log records that. SafeGrd deletes the URL when the restore ends, or after a day if nothing picked it up. - To put a live database back, restore into a new database beside it and point your application at that. Or restore only the broken schemas or tables into the live database. ### A backup before you restore into a live database When you restore chosen schemas or tables, they go into a database that has other data in it, usually the live one. So SafeGrd backs up the surface first, and you can go back to how it was. If that backup fails, the restore does not run. A backup taken in the last 15 minutes counts as that backup. You can turn this off in the dialog. ### The record of a restore The **Restores** tab lists who asked, what was restored into which target, and how many tables and rows came back. SafeGrd signs each completed restore with its attestation key, and **Certificate** shows the signed record. A restore is not a Fire Drill, so it does not count toward your plan's drills, the badge or the recovery point. ### What a PostgreSQL restore turns off and takes out - `pg_cron` jobs come back turned off. The restore prints each one with its id and the statement that turns it on, so nothing runs against restored data until you say so. To keep them as they were, pass `--keep-cron-active`. See [pg_cron jobs](https://safegrd.dev/docs/surfaces/postgres#pg-cron). - For the tables a surface names under `erasures:`, erased rows stay erased. Before it commits, the restore deletes each row that was deleted after the backup was taken, and prints how many. If it cannot read the erasure record, it restores nothing. To restore anyway, pass `--without-erasures`. See [the erasure record](https://safegrd.dev/docs/surfaces/postgres#erasures). ## Fire Drills A Fire Drill restores a snapshot somewhere disposable, counts what came back and records the result. It runs in _your_ environment with _your_ key. SafeGrd's server receives the report, not the data. ``` # in-memory dry restore, no database needed safegrd verify --snapshot snap-1b094c0af8 # full drill into an ephemeral database safegrd verify --snapshot snap-1b094c0af8 --sandbox-target "$SANDBOX_URL" ``` By default, `verify` does a dry run. It decrypts the stream, replays it in memory and checks table counts, row counts, column counts and extensions, with no PostgreSQL running anywhere. It checks the data but does not load the schema. Only a sandbox drill does that. Because the dry run needs no database, you can run it in CI or on a laptop. For the full drill, pass `--sandbox-target` without `--dry-run`. The drill loads the data into a throwaway database and checks it there. SafeGrd records sandbox drills on every paid plan. On the free plan, it records the in-memory drill and refuses a sandbox drill, with the reason. After loading the data, a sandbox drill has the database check it: - PostgreSQL runs `amcheck` over every table (PostgreSQL 14 and later) and every B-tree index. - MySQL runs `CHECK TABLE` on every table. - MongoDB runs `validate` on every collection. - SQLite runs `PRAGMA integrity_check`. The result is one line of the drill and part of its certificate. Sometimes the sandbox cannot run the check: PostgreSQL has no `amcheck`, the role may not create it, or a MongoDB user has no `dbAdmin` on the sandbox. Then the drill says `NOT RUN` with the reason, and passes or fails on its other checks. To have PostgreSQL's check run, let the sandbox's role run `CREATE EXTENSION amcheck`, or create the extension in the sandbox database yourself. A database drill record also shows how each table's row count changed since the previous backup. It lists the tables that grew or shrank, largest change first, then the tables added and removed, and how many did not change. The console shows this in the drill's detail, and it is signed with the record ([how](https://safegrd.dev/docs/attestation#table-changes)). Every plan runs scheduled Fire Drills. The free plan drills once a week, in memory. Paid plans drill weekly or daily and record sandbox drills ([pricing](https://safegrd.dev/pricing)). A surface's Details show when its next backup and drill are due, and who starts them. SafeGrd starts the ones it runs on time. A host's daemon runs a due one at its next check-in. For a backup from cron or a CI workflow, that schedule decides the time, and the console shows the earliest time the next drill can run. _Upcoming_ on the Fire Drills tab lists every surface's next drill, overdue ones first, with where it runs and how often the plan drills. ### Your own checks Row counts show the data came back. Your own checks show it is the data you expect: no user without an email, orders from the last day, a migration table at the right version. A sandbox drill runs each check after the restore and records its result beside the counts. In the console, add queries under _Drill checks_ on a database surface, each with the value it must return. The host's daemon picks them up at its next check-in, and a drill on SafeGrd's machines runs them too. In the host's config, put the same queries under `drill.checks`. The config also takes commands: ``` surfaces: - id: app-db drill: checks: - name: every user has an email sql: SELECT count(*) FROM users WHERE email IS NULL expect: "0" - name: orders in the last day sql: SELECT count(*) FROM orders WHERE created_at > now() - interval '1 day' expect: "> 0" - name: newest order is as new as the backup sql: SELECT max(created_at) >= timestamptz '${SAFEGRD_SNAPSHOT_CREATED_AT}' - interval '1 hour' FROM orders expect: "true" - name: app smoke test command: ./scripts/smoke.sh ``` - A query runs in a read-only transaction, for at most 60 seconds. The first column of its first row is compared with `expect`, which is a value (`"0"`, `"true"`, `"ok"`) or a comparison (`"> 0"`, `">= 1000"`, `"!= 0"`). Numbers compare as numbers, so `42.75` matches a `numeric` column. Queries run on PostgreSQL, MySQL, MariaDB and SQLite sandboxes. - In a query, `${SAFEGRD_SNAPSHOT_CREATED_AT}` is replaced with the time the backup was taken, in RFC 3339 and UTC to the second (`2026-10-20T09:30:00Z`). Quote it. Compare your newest row with it to fail a drill of a backup whose data had stopped changing. Row counts alone do not catch that. The console's _From a template_ writes this check for the surface's database: PostgreSQL: `timestamptz '${SAFEGRD_SNAPSHOT_CREATED_AT}'` - MySQL and MariaDB: `CAST('${SAFEGRD_SNAPSHOT_CREATED_AT}' AS DATETIME)`, which is UTC, so compare it with `UTC_TIMESTAMP()` or a column written in UTC - SQLite: `datetime('${SAFEGRD_SNAPSHOT_CREATED_AT}')`, and `datetime(created_at)` on the column too, since text compares letter by letter Any other `${SAFEGRD_...}` in a query is refused when you save the check. - A command can be set only in the host's config, not in the console, and it never runs on SafeGrd's machines. It runs through `sh -c` on the host that runs the drill, for at most 5 minutes, and passes if it exits 0. It gets these environment variables: `SAFEGRD_SANDBOX_URL`, the sandbox's connection string - `SAFEGRD_SNAPSHOT_ID` - `SAFEGRD_SNAPSHOT_CREATED_AT`, when the backup was taken (RFC 3339, UTC). Compare your newest row with it to fail a drill of a backup whose data had stopped changing. On a MongoDB sandbox, write each check as a command. When a command fails, the drill record keeps the last 1,000 characters it printed, with the sandbox's password replaced. - `timeout_seconds` on a check changes its time limit. A check that does not hold fails the drill. The alert names the check and says the restore matched the backup. The certificate hash covers each check's name, result and expected value, so the signed record fixes what every check returned. An in-memory drill has no database to run the checks against. It runs without them and prints a warning with how many it skipped. On a plan with sandbox drills, the daemon drills a SQLite surface that has no `drill.sandbox_url` into a new file in its state directory, so its checks run, and deletes the file afterwards. `safegrd doctor` reads every surface's checks and reports any that cannot run. For a single drill by hand, put the same list in a file: ``` safegrd verify --snapshot snap-1b094c0af8 --sandbox-target "$SANDBOX_URL" --checks checks.yaml ``` ### Read rows out of a backup To get back a few rows without restoring the table they came from, use `--query`. It restores a database snapshot into a sandbox, runs one query there in a read-only transaction, and prints the rows as CSV. Column names come first, and NULL is an empty field. `--out` writes the CSV to a file instead of stdout. ``` safegrd verify --snapshot snap-1b094c0af8 --query "SELECT * FROM orders WHERE id = 4211" --out order.csv ``` With `--sandbox-target`, the restore goes into that empty database and stays there. Without it, a PostgreSQL snapshot is restored into a cluster that the command starts on the host and removes afterwards. That needs `initdb` and `postgres` of the snapshot's version or newer. `--keep` leaves the cluster running for a day and prints how to connect. A query run is not a drill. Nothing is reported to SafeGrd, and the MCP server does not offer it. A full drill of a demo database, as `safegrd verify` prints it: ``` Warning: the value of --sandbox-target is visible in `ps` and shell history. Pass env:VAR or file:/path instead. Verifying snap-20261031-135407-16c6d3 (epoch e202610-b3036e43, shop-db): checking the packs, restoring it into postgres://postgres:xxxxx@localhost:62309/drill_sandbox?sslmode=disable. [PASS] DecryptionIntegrity (Expected: the run, its indexes and trees open with the identity, Actual: they do) [PASS] RepositoryIntegrity (Expected: every pack present and agreeing with its index, Actual: 3 packs checked) [PASS] DigestIntegrity (Expected: f9eef402a29aca7aac7575ff9de386b4e31978b155fc35b5affd11e1923133b1, Actual: f9eef402a29aca7aac7575ff9de386b4e31978b155fc35b5affd11e1923133b1) [PASS] TableCountMatch (Expected: 4 tables, Actual: 4 tables) [PASS] RowCountMatch (Expected: 45404 rows, Actual: 45404 rows) [PASS] Row Count: public.customers (Expected: 1840 rows, Actual: 1840 rows) [PASS] Row Count: public.order_items (Expected: 31000 rows, Actual: 31000 rows) [PASS] Row Count: public.orders (Expected: 12500 rows, Actual: 12500 rows) [PASS] Row Count: public.products (Expected: 64 rows, Actual: 64 rows) [PASS] Newest Row: public.customers (Expected: created_at 2026-10-31T13:53:57.447761Z, Actual: created_at 2026-10-31T13:53:57.447761Z) [PASS] ExtensionBootCheck (Expected: [], Actual: []) [PASS] Schema Captured By pg_dump (Expected: schema from pg_dump, Actual: pg_dump 18.6) [PASS] Data Check: amcheck (Expected: every heap and B-tree index readable, Actual: 4 tables, 3 B-tree indexes, no corruption) Fire Drill Passed Verification ID: verif-cc68f9a0 Duration: 236ms Tables: 4 Rows: 45,404 Certificate: cert_sg_d05260a52f24625bb27f4687d67b67781d72da6b13b2003c33128037d1f88ecf ``` ## Drills on SafeGrd If a surface has no host to drill it, for example one you back up from a GitHub Actions workflow, SafeGrd can run its drills. Turn on _Drill on SafeGrd_ on the surface. Each drill runs on a machine SafeGrd starts for it and destroys afterwards, and the record is the same one a host sends. [Hosted backups and drills](https://safegrd.dev/docs/hosted) covers which surfaces can use it, what each machine is given, and how minutes are counted. ## Recovery point and restore time For each surface, the console shows four figures, measured from its snapshots and drills. The first three measure the recovery point. A security questionnaire compares that with your recovery point objective (RPO), and restore time with your recovery time objective (RTO). SafeGrd sets no target and guarantees none. [RPO and RTO for a database backup](https://safegrd.dev/guides/rpo-rto) explains both. | Figure | How it is measured | | --- | --- | | Verified | The time since the newest snapshot that a drill restored, and how deep that drill went. The console leads with this one, because it counts only backups that came back. | | Latest backup | The time since the newest snapshot that was written. A snapshot that Threat Shield flagged still counts, because the data is there. The console marks it. | | Worst gap, 30 days | The recovery point at the surface's worst moment in the last 30 days: the longest time between two backups in those days. It counts from the last backup before the window and includes the time since the newest one. A surface backed up for less than 30 days is measured from its first backup, and the console says so. The gap is shown against the schedule. A gap over twice the schedule is red, because that is when the overdue alert fires. Time the schedule was paused in the console is left out of each gap, and the console says how much. The recovery figures and the Evidence Pack list each pause with who paused it and why. | | Restore time | The median (p50) and 95th percentile (p95) duration of this surface’s passed drills in the last 90 days. Next to them, the console shows how many drills that is, the usual size of the snapshots they restored, and where they ran. | A drill is timed from before the download, through decryption, to the end of the restore. Restore time counts only drills that wrote the data out: a sandbox drill, which loads a database, and a restore of a repository, which writes the files. In-memory drills are left out, because they take less time than any real restore. For a database, an in-memory drill checks the backup without loading the data or building an index. For a files or mail archive, it reads the archive without writing a file. SQLite is the exception. Its in-memory drill writes the database file out and opens it, which is all a SQLite restore does, so it counts. If a surface has no drill that counts, the console shows why instead of a number. Either the free plan drills in memory, the host could not run a sandbox drill (the console names the reason), or no drill has passed yet. Drills that ran on SafeGrd’s machines are counted separately from drills on your host. Those machines are not yours, so a drill can take a different time there than a restore on your own hardware. To get the time on your hardware, time a restore yourself: [Rehearse it before you need it](https://safegrd.dev/docs/runbooks#rehearsal). The same figures are in `GET /api/v1/orgs/{org_id}/recovery` and in the Evidence Pack, each with the date it was measured. ## Verifying from another machine You can run a drill on a machine other than the one that took the backup. Point `verify` at the storage directly and give it the key: ``` safegrd verify --snapshot snap-1b094c0af8 \ --s3-bucket my-worm-bucket --s3-endpoint https://s3.example.com \ --s3-access-key "$AWS_ACCESS_KEY_ID" --s3-secret-key env:AWS_SECRET_ACCESS_KEY \ --key-path ./daemon.key ``` This is what disaster recovery looks like: a separate machine with read access to the bucket and your key rebuilds the data from nothing. [The recovery runbooks](https://safegrd.dev/docs/runbooks) take you through it step by step. [Testing that a backup restores](https://safegrd.dev/guides/test-database-backup-restore) explains the same checks with plain `pg_restore` and `mysql`. You can also drill another host's snapshot from any host enrolled in the same organization. Run `safegrd verify --snapshot` there, and the console records the result as a Fire Drill on the other host's surface. The drilling host can be in any project, and it can be one enrolled only to restore. It needs the snapshot's key. SafeGrd releases a SafeGrd-managed key to it. A customer-managed key has to be on that host already. If you keep your customer-managed key on a second host this way, that host can drill the other host's backups too. --- Source: https://safegrd.dev/docs/runbooks # Recovery runbooks Follow these steps in order on the day you need them, even if you have never run `safegrd` before. An automated test runs every command here, in this order, on each change to SafeGrd. The [first runbook](https://safegrd.dev/docs/runbooks#managed) runs on a new host enrolled into the same project. The others run on a machine that has only the key file and read access to the bucket. Which one you follow depends on the host's key, which the console shows on each host. With a SafeGrd-managed key, start at [the host is gone](https://safegrd.dev/docs/runbooks#managed). With a customer-managed key, start at [before you start](https://safegrd.dev/docs/runbooks#before). If you cannot reach SafeGrd, start at [without SafeGrd](https://safegrd.dev/docs/runbooks#without-safegrd). These steps need a CLI newer than v0.0.2. Earlier versions cannot find a snapshot without its node id. Check with `safegrd --version`, and install the latest if yours is older. ## With a SafeGrd-managed key: the host is gone A SafeGrd-managed key is the default. SafeGrd keeps each host's key sealed and releases it only to your organization's enrolled hosts, so a new host can read what the old one wrote, even after you remove the old host from the console. You need a machine that can reach the database or directory you are restoring into, and a login to the console. You do not need a key file or bucket credentials. - Install and enroll the new machine: ``` curl -fsSL https://safegrd.dev/install.sh | sh ``` Answer `Y` to connect it, then approve the sign-in in a browser on any device. If the organization has more than one project, enrollment asks which one. Choose the old host's project, so the new host uses the same storage. Without a terminal, the machine joins the default project. In that case, run `safegrd enroll --project SLUG` instead (the slug is on the console's Projects page). - Find the snapshot to restore with `safegrd list`, or on the console's Snapshots page, where each row names the surface and when it was taken. Copy its ID. - Create a new, empty database and restore into it: ``` safegrd restore --snapshot SNAPSHOT_ID --target "postgres://user:pass@host:5432/recovered" ``` For a directory or a mailbox, use `--target-dir ./restored`, as in [Runbook 2](https://safegrd.dev/docs/runbooks#files) and [Runbook 3](https://safegrd.dev/docs/runbooks#mailbox). The restore prints _Decryption key: SG:…, SafeGrd-managed key of org … (fetched, not stored)_. That means SafeGrd released this snapshot's key to this host for this restore, and the host did not write it to disk. Before it reports success, the restore checks every table against the digest and row count recorded at backup time. - Check what was released, and to which host: in the console, Projects, the project's _Storage_, under _Key access record_. The row names this host, the key it was given and the snapshot. - Back up the restored database from this host: in the console, click _Add surface_, then run the command it shows on this machine. - Retire the old host: under Hosts, open its _Details_, then _Edit_, and choose _Retire_. Its snapshots stay in storage, and SafeGrd keeps its key, so they still restore. Until you retire it, the console marks it _Silent_ after three missed check-ins (at least 15 minutes) and alerts the organization's owners. The same steps work when the project's backups go to your own bucket (set under the project's _Storage_ in Projects). SafeGrd gives the enrolled host the bucket's credentials the same way. ## Restoring without SafeGrd A restore needs the snapshot's bytes and its key. The CLI can read both with no SafeGrd account or server, so what you do depends on where they are. | Backups in | What to do | | --- | --- | | Your own bucket | Follow [before you start](https://safegrd.dev/docs/runbooks#before) and [set up the recovery machine](https://safegrd.dev/docs/runbooks#recovery-config), then the runbook for what you lost. | | Hosted storage | Keep a copy in your own bucket with `safegrd export` (below), and restore from that copy the same way. Hosted storage is read and written through SafeGrd: each upload gets its URLs from SafeGrd's server, so backups to hosted storage run while the server answers. A backup to your own bucket needs only the bucket. | With either kind of key, enrollment writes the key file to `~/.safegrd/keys/daemon.key` on the host that took the backups. Keep a copy of it with your copy of the backups. ### Keep a copy of hosted backups Run `safegrd export` on the host on a schedule. It copies each snapshot still encrypted, checks it against the digest recorded at backup time and keeps its lock date. It skips snapshots already in the bucket, so each run copies only what is new. The bucket needs Object Lock (or pass `--to-worm-mode NONE`). The credentials come from the standard AWS environment variables. A daily crontab line: ``` 0 3 * * * AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... safegrd export --to-bucket your-bucket --to-endpoint https://your-s3-endpoint ``` Or let the daemon run it, with no crontab line. Name the bucket in `daemon.export_to` in the host's config, and the daemon copies to it daily after its backups. The console shows on the host when the last copy was made. The credentials take `env:VAR` or `file:/path`. ``` daemon: export_to: bucket: "your-bucket" endpoint: "https://your-s3-endpoint" access_key_id: "env:EXPORT_ACCESS_KEY_ID" secret_access_key: "env:EXPORT_SECRET_ACCESS_KEY" schedule: "@daily" ``` Leave out `--to-endpoint` (or `endpoint`) on AWS. The copy is filed under `safegrd/snapshots`, the `prefix` the recovery config below uses. `--to-dir` copies to a directory instead, and [Runbook 4](https://safegrd.dev/docs/runbooks#from-export) restores from that. ## With a customer-managed key: before you start You need three things: - The private key file, which starts `AGE-SECRET-KEY-1`. It is usually `~/.safegrd/keys/daemon.key` on the host that took the backups. Keep a copy offline. With a customer-managed key, it is the one file that decrypts these backups. - Read access to the bucket: its name, its region, the endpoint if it is not AWS, and an access key that can list and read it. - The `safegrd` CLI on the machine you are restoring to: `curl -fsSL https://safegrd.dev/install.sh | SAFEGRD_NO_SETUP=1 sh`. `SAFEGRD_NO_SETUP=1` installs the binary and stops there. Enrolling this machine would generate a new key, and a restore needs the old one. You do _not_ need SafeGrd, the old host or its config. If the host that took the backups still works, skip the recovery config below and run the same commands there. ## Set up the recovery machine - Copy the key file to this machine and make it readable only by you: ``` chmod 600 ./safegrd.key ``` - Find the bucket's `prefix`. With a wrong prefix, the CLI finds no snapshots at all. If the bucket was set up in the console, the old host's `config.yaml` does not have it, so read it from the bucket. An incremental backup is stored under `PREFIX/NODE_ID/repo/`, and a full copy as `PREFIX/NODE_ID/SNAPSHOT_ID.safegrd`. The prefix is everything before the node id. It is `safegrd/snapshots` unless someone changed it. ``` aws s3 ls s3://your-bucket/ --recursive | grep -E '/repo/|\.safegrd$' | head -3 ``` Add `--endpoint-url https://your-s3-endpoint` when it is not AWS. Write the prefix down now, while you can also check it under the project's _Storage_ in Projects. - Write `recovery.yaml`. Leave out `endpoint` on AWS. Set `force_path_style: true` only for MinIO. ``` storage: type: "s3" bucket: "your-bucket" region: "us-east-1" endpoint: "https://your-s3-endpoint" prefix: "safegrd/snapshots" encryption: key_path: "./safegrd.key" ``` safegrd reads a config only if no one but its owner can read it: ``` chmod 600 recovery.yaml ``` - Pass the bucket credentials through the environment, so no secret sits in the file: ``` export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... ``` - List what is there. Each row names the snapshot, the host it was filed under, what it holds and when it can be deleted. Incremental file backups appear in a second table, by month. You do not need to know the node id, because restore finds it. ``` safegrd --config recovery.yaml list ``` - Before you touch anything, check that the snapshot you picked decrypts and is whole. This writes nothing anywhere: ``` safegrd --config recovery.yaml verify --snapshot SNAPSHOT_ID --in-memory ``` If it fails, stop. Try the next-newest snapshot, and see [Troubleshooting](https://safegrd.dev/docs/troubleshooting). ## Runbook 1: the database is gone - Create a new, **empty** database to restore into. Do not restore over a database that still holds data you may need. Restore into a new one, check it, then switch your application over. ``` createdb recovered ``` - Restore into it: ``` safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target "postgres://user:pass@host:5432/recovered" ``` Before it reports success, the restore checks the snapshot's digest. If it could check it only against the sidecar, it says so, which is expected when SafeGrd is unreachable. It creates any role the schema names that the new cluster lacks, with its password, and prints each one. Run it as a superuser so every owner and grant comes back ([roles, owners and grants](https://safegrd.dev/docs/surfaces/postgres#roles)). - Check the tables and row counts it printed against what you expect, then run a query your application depends on. - Point your application at `recovered`. Once it is serving, take a fresh backup from it: `safegrd backup --database-url ...` ## Runbook 2: a directory tree is gone - Restore into a **new** directory, not over the old path: ``` safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target-dir ./restored ``` - Read the last lines it prints, which say what it restored and what it cannot restore. Ownership comes back only when you run the restore as root. Hard links, extended attributes and ACLs are not restored ([the full list](https://safegrd.dev/docs/surfaces/files#restore)). - If you need ownership back, run the same command with `sudo`, pointing at the same config and key. - Check a few files you know, then move the directory into place. A restore never writes outside `--target-dir`. If the target already has a symlink where the backup has a file or directory, the restore stops and names the symlink instead of following it. ## Runbook 3: a mailbox is gone - Restore the messages to disk: ``` safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --target-dir ./mail ``` - Each mailbox folder becomes a directory and each message a standard `.eml` file, readable only by you. Any mail client can open them. A Gmail mailbox comes back as one `All Mail` directory, with each message's labels in `.safegrd-email-manifest.json` ([Gmail labels](https://safegrd.dev/docs/surfaces/email#gmail-labels)). - To put them back on a server, import the directory with your mail system's own import tool, or drag the `.eml` files into the mailbox from a desktop client connected over IMAP. ## Runbook 4: restore from an export You can restore from a `safegrd export --to-dir` copy with only the key, and no bucket. The export writes a single-archive snapshot as `NODE/SNAPSHOT_ID.safegrd`, with its metadata beside it in `SNAPSHOT_ID.meta.json`. It writes an incremental backup (the default) as its month of packs under `NODE/repo/`. The snapshot ID is the one `safegrd backup` printed, or the name of a file under `snapshots/` in the export. - Restore from the export directory, or from one file in it: ``` safegrd restore --from ./export --snapshot SNAPSHOT_ID --target-dir ./restored safegrd restore --from ./export/NODE/SNAPSHOT_ID.safegrd --target-dir ./restored ``` The key comes from `--key-path`, `--private-key` or the config, as in the runbooks above. The restore checks the digest against SafeGrd's record when the host can reach SafeGrd, and against the `.meta.json` beside the file when it cannot. - For a database snapshot, use `--target` instead of `--target-dir`, as in [Runbook 1](https://safegrd.dev/docs/runbooks#database). ### Without SafeGrd A `.safegrd` file is zstd-compressed data encrypted to your Age key, so [age](https://age-encryption.org) and zstd open it: ``` age -d -i safegrd.key SNAPSHOT_ID.safegrd | zstd -d > payload ``` For a files or email snapshot, `payload` is a tar archive. `tar -xf payload` gives back the directory tree, or the `.eml` files and their manifest. A PostgreSQL snapshot is a tar archive too: the schema as SQL from `pg_dump`, then each table's rows in PostgreSQL's binary `COPY` format. `pg_restore` does not read it, but `psql` can load it once it is written out as SQL files. `safegrd restore --snapshot ID --to-sql DIR` writes those files, with a `load.sql` that runs them in order. `psql -f load.sql` then loads them into an empty database with no SafeGrd involved. This route checks nothing, so compare `sha256sum payload` with `sha256_checksum` in the `.meta.json` yourself. It works only for single-archive snapshots: mailboxes, and databases or files backed up with `format: tar`. An incremental backup is a set of packs, indexes and directory listings, not one archive. For those, `safegrd export --to-dir` copies a whole month, and `safegrd restore --from` reads the copy with the key and no bucket. ## Runbook 5: one file, as it was on a given day Use this for a directory tree backed up incrementally, the default for files. To restore the whole tree, follow [Runbook 2](https://safegrd.dev/docs/runbooks#files). - List the kept versions of the file. Paths are relative to `/`, with no leading slash: ``` safegrd --config recovery.yaml find etc/nginx/nginx.conf ``` Each version is one distinct content of the file, numbered from the oldest, with the first and last snapshot that holds it. For a directory, `find` lists everything below it. `find --deleted var/www/uploads/` lists only files the newest backup no longer holds. - Restore the version whose dates bracket the day you want, into a new directory: ``` safegrd --config recovery.yaml restore --path etc/nginx/nginx.conf --version 2 --target-dir ./restored ``` Or restore everything under a directory as it was in one snapshot, with the id from `find` or `list`: ``` safegrd --config recovery.yaml restore --snapshot SNAPSHOT_ID --path 'var/www/uploads/**' --target-dir ./restored ``` The file comes back under the target at the path a whole-tree restore would give it (from the root for a surface of one directory, from `/` otherwise). It keeps the permissions and times it had in that snapshot. The restore downloads only that file's chunks and checks every byte against the digest recorded at backup time. - Compare `sha256sum` of the restored file with the SHA-256 `find` printed, then copy it into place. ## Rehearse it before you need it Rehearse a restore every quarter, and before a compliance audit, following this page: - Use a machine that is **not** your production host, such as a laptop or a fresh VM. - Put only the key file and the read-only bucket credentials on it, and follow [Set up the recovery machine](https://safegrd.dev/docs/runbooks#recovery-config). - Run [Runbook 1](https://safegrd.dev/docs/runbooks#database) into a scratch database, [Runbook 2](https://safegrd.dev/docs/runbooks#files) into a scratch directory, and [Runbook 5](https://safegrd.dev/docs/runbooks#one-file) for one file from a month ago. - Write down the date, the snapshot id and how long it took. That is your real recovery time. A Fire Drill proves each snapshot restores, and this rehearsal proves that _you_ can restore it. The console’s [restore time](https://safegrd.dev/docs/verify#recovery) comes from drills, some of them run on SafeGrd’s machines. This one is timed on your own hardware. - Delete the scratch copies, and remove the key file from the machine if it is not kept secure. --- Source: https://safegrd.dev/docs/security # Security and key custody Your host encrypts every backup before it uploads it. The bucket keeps it under S3 Object Lock until a date set by your plan or your retention settings. Unless you choose a customer-managed key, SafeGrd keeps the key that decrypts your backups sealed and releases it only to your enrolled hosts. ## The cryptography | Layer | What is used | | --- | --- | | Encryption | Age, which uses ChaCha20-Poly1305 for authenticated encryption and X25519 for key agreement. It is a standard scheme you can audit, with nothing bespoke. | | Compression | zstandard, applied before encryption on the streaming path. | | Attestation | Ed25519 signatures over a SHA-256 hash chain. | | Implementation | Written in Go, with no wrapped proprietary blobs. Database surfaces use the engine's own client tools: `pg_dump` for a PostgreSQL schema, `mysqldump` and `mysql` for MySQL and MariaDB, `mongodump` and `mongorestore` for MongoDB. SQLite, files and mail need no other tool. | Encryption is asymmetric, so a host needs only the _public_ key to back up. You need the private key only to restore or to run a Fire Drill. The CLI sets no server-side encryption header when it uploads, because the object is already Age ciphertext. Your bucket's own encryption at rest (SSE-S3, which AWS applies to every new bucket) is up to you. It adds nothing to a backup's confidentiality. ### Data keys and key-encryption keys In envelope-encryption terms, every backup has its own data encryption key (DEK), and your age key pair is the key-encryption key (KEK) that wraps it. | Key | What it does | Where it lives | | --- | --- | --- | | DEK | Age makes a random 16-byte file key for each backup, and encrypts the backup with ChaCha20-Poly1305 under a key derived from it. | In the backup's header, wrapped by the KEK. Each backup gets a new one, so there is nothing to store or rotate. | | KEK | Your age key pair (X25519). The public half wraps each backup's DEK. The private half unwraps it for a restore or a drill. | The public half is on your hosts and in SafeGrd. Where the private half lives depends on your [key custody](https://safegrd.dev/docs/security#two-kinds) choice: a SafeGrd-managed key or a customer-managed key. | With a SafeGrd-managed key, SafeGrd keeps your key sealed and releases it only to your enrolled hosts, so you can restore even after losing a host. With a customer-managed key, the private half of the KEK stays on your hosts. The KEK is the only key you make a choice about. A SafeGrd-managed key works like a key your cloud provider manages for you. SafeGrd keeps it per organization, releases it only to your enrolled hosts and audits every release. A customer-managed key never reaches SafeGrd. In cloud terms, that is client-side encryption rather than a key in a provider's KMS. ## What SafeGrd's server never receives - Your data. Your host sends the backup to the bucket already encrypted. It never passes through SafeGrd's server, even when the bucket is hosted storage. A drill you set to run on SafeGrd decrypts the snapshot on a machine SafeGrd starts for that drill alone. The machine gets the key and never a database credential. The release shows in your key access record, and the machine is destroyed when the drill ends. When SafeGrd backs up a database for you, a machine started for that backup alone runs the dump. It gets the connection string and never the key. - Filenames, table contents and email subjects. The manifest is sealed inside the encrypted archive, not written beside it. - Your age private key, if it is a customer-managed key. The CLI never writes it into the config file and never sends it anywhere. - Database and mailbox passwords, unless you choose to have SafeGrd hold them (below). ## Two secrets, each chosen separately SafeGrd needs two secrets to protect a surface. You choose for each one whether SafeGrd keeps it, and choosing SafeGrd for one does not choose it for the other. | Secret | What it does | Default | | --- | --- | --- | | The encryption key | The age private key that decrypts a host's backups (the private half of the [KEK](https://safegrd.dev/docs/security#dek-kek)). Restores and drills need it. Backing up needs only its public half. | A SafeGrd-managed key, kept sealed by SafeGrd. You choose per host when it enrolls. `--key-custody local` keeps it on the host as a customer-managed key. | | A surface's credential | The database URL or mailbox password the host uses to read the surface, or the S3 key of a bucket SafeGrd backs up. Only backups need it. | The host keeps it. You choose per surface: the setup wizard asks, or use _Credential_ in the surface's panel to hand it to SafeGrd. | A bucket's access key belongs to the project rather than to a host. If you set the bucket up in the console, SafeGrd keeps the access key sealed. If you set it up with the CLI, the hosts keep it. Hosted storage needs no key of yours. ### The encryption key You choose each host's key mode when the host enrolls, with `--key-custody`. In both modes the host encrypts every backup the same way before it leaves. The difference is where the private key lives, and so how a new host gets it. SafeGrd-managed key --key-custody safegrd, the default - This host makes the key when it enrolls - SafeGrd keeps it sealed - Your enrolled hosts get it to restore or drill What it gives you SafeGrd keeps your key sealed and releases it only to your enrolled hosts, so you can restore even after losing a host. Every release is written to the key access record before it happens. A copy [`safegrd keygen --held`](https://safegrd.dev/docs/cli#keygen) writes one on an enrolled host, if you want your own. Customer-managed key --key-custody local - This host makes the key and keeps it - Your copy somewhere safe, such as a password manager - A new host you copy the key file to it What it gives you Only you can decrypt these backups. SafeGrd has the public key and nothing else. Keep a copy of the key file somewhere safe, before the first backup. You lose no backups when you lose a host, in either mode. With a SafeGrd-managed key, enroll a new host and restore ([the runbook](https://safegrd.dev/docs/runbooks#managed)). With a customer-managed key, put your copy of the key file on the new host first. The console shows how each surface's key is managed. The CLI never sends a key you supply yourself. When SafeGrd backs up a database or bucket for you, there is no host to make a key. Those backups are sealed to a key SafeGrd makes and keeps for your organization. ### Credentials When SafeGrd holds a credential, it keeps it sealed and releases it only to the surface's host when it backs up, or to the machine SafeGrd starts for that backup. Nobody can read it back through the console or the API. A credential you keep on the host stays in its config, which points to it with an environment variable, a command or a file. To restore from the console you need an account that can write, and the backup's own account cannot. Type that account's URL when you ask for the restore. SafeGrd seals it and gives it once to the host or the machine SafeGrd starts that runs the restore. The release is recorded in the key access log. SafeGrd deletes the URL when the restore ends, or after a day if nothing picked it up. A machine SafeGrd starts for a restore gets the snapshot's key and that one URL, for that restore only. It never receives the surface's own credential. Whichever you pick, the archive is sealed the same way. [Install and enroll](https://safegrd.dev/docs/install#custody) shows how to choose the key when a host enrolls. ### SafeGrd's own database SafeGrd's database holds your account, your hosts' enrollment, the record of every backup and drill, and the sealed secrets above. ## If a host, a token or a key file is compromised Work through the steps in order. The first one cuts off access, and the later ones show what was reached. Each step names the console action or the command to run. | What happened | What to do | | --- | --- | | A host is lost or compromised | Rotate the host's token: Hosts, the host's _Details_, _Edit_, _Rotate token_. The old node token stops working at once. The host can no longer report, fetch a held credential or receive the key. Rotate the surface's credential at the database or mail provider, whether SafeGrd or the host kept it. Whoever had the host could read its config. If SafeGrd holds the credential, save the new one from _Credential_ in the surface's panel. Check what the host was given: Projects, the project's _Storage_, _Key access record_. It lists every release of a held secret to that host, with the time. That is what the host could have decrypted or connected to. Enroll the replacement with `safegrd enroll`. With a SafeGrd-managed key it can restore at once ([runbook](https://safegrd.dev/docs/runbooks#managed)). With a customer-managed key, copy the key file to it. | | A personal access token leaked | Revoke it: Tokens, the token, _Revoke_. SafeGrd refuses every request with it from then on. A token can enroll a host, read, and ask for a backup or a drill. Check Hosts for a host you did not enroll, and the _Key access record_ for a release to one. _Activity_ in the console shows what changed while the token was out. Create a new token for whatever used the old one, and pass it as `env:VAR` or `file:/path`. | | An agent's sign-in token leaked | Revoke it: Tokens, the row that says _Signed in from_ and the client's name, _Revoke_. The client's access and refresh tokens stop working on their next call. A sign-in token works only on the MCP server, and the REST API refuses it. Its access token lasts an hour. The refresh token that renews it is single-use, and presenting one twice ends the whole sign-in on its own. While it was out, it could do what a personal access token can: read, and ask for a backup or a drill. Connect the client again. It signs in afresh. | | A customer-managed key file was copied | Rotate the bucket's access key at your storage provider, and save the new one under the project's _Storage_ in Projects. Someone with the key file can read a backup only if they can also download it. On the host, run `safegrd init --force`. It writes a fresh config and a new key, so SafeGrd sees the host as a new one. Run `safegrd enroll` to send the new public key, then add its surfaces again. Backups from then on are encrypted to the new key. Keep the old key file until the earlier snapshots expire, because they still open with it. Guard it as before: anyone holding it and a copy of a snapshot can read that snapshot. | ## The activity log _Activity_ in the console is the audit log for owners and admins. Each row names who did what, and when. It records: - sign-ins, and sign-ins refused for a wrong password or two-factor code - members invited, joined and removed, and their roles changed - access tokens created and revoked - hosts enrolled and decommissioned, and their tokens rotated - surfaces added and retired, and their connections changed - changes to projects and storage - sign-in rules and two-factor turned on or off - alert and compliance settings Every release of a held key or credential appears here too, as well as in the _Key access record_. A host reading its own hosted storage appears only in the key access record. Owners and admins can read both and export them as CSV or JSON (`GET /api/v1/orgs/{id}/events/export?category=action|access&format=csv|json`). SafeGrd keeps them for 90 days, so export the log to keep it longer. ## Check these claims in the CLI's source You can check everything this page says about what leaves your host in the CLI's source, at [github.com/safegrd/cli](https://github.com/safegrd/cli). It is licensed under the **Business Source License 1.1**, so you can read, modify, build and evaluate it. Any SafeGrd account comes with production use, the free plan included. Without an account you can still list, verify, restore and export your backups. If SafeGrd stops running its service, all production use is allowed. Each version becomes GPLv3 four years after its release ([license](https://github.com/safegrd/cli/blob/main/LICENSE)). ## Hardening the host - Keep `config.yaml` and the key file at 0600. On POSIX systems `safegrd doctor` fails when they are not. - Give the backup credential read-only database access. A backup never writes to the database. - Deny `s3:DeleteObject` to the backup credential. Object Lock already stops a locked backup from being destroyed, and denying deletes keeps the credential from writing delete markers at all. - Use a `credential` block (`from: env`, `command` or `file`) instead of an inline `database_url` with a password in it. That keeps the password out of the config, shell history and the process table. - Before you need it, run a real restore at least once on a host that is not production. --- Source: https://safegrd.dev/docs/threat-shield # Threat Shield Threat Shield tells you when a backup succeeded but the data in it shrank, and which earlier snapshot is the known-good one to restore from. A backup that exits 0 can still hold the wrong data. If a migration drops a table or a script truncates one, the next backup copies what is left and reports success. When the host reports the backup, Threat Shield compares the snapshot's counts with the one before it. If they dropped sharply, the snapshot is marked anomalous instead of healthy. ## What it compares Threat Shield does not open snapshots. It works from the counts the host reports with each backup, and those say nothing about the contents: - Tables and rows per table (Postgres, MySQL, MariaDB and SQLite) - Each table's columns, by name (Postgres, MySQL, MariaDB and SQLite) - Collections and documents (MongoDB) - Installed extensions (Postgres) - Files (file trees) - Messages (mailboxes) ## What triggers an anomaly | Anomaly | Severity | Condition | | --- | --- | --- | | row_volume_drop | Critical | Compared with the snapshot before, a database's total rows drop by more than 20%, a file tree's files by 25% or more, or a mailbox's messages by 30% or more. Rows that left with an old partition are not counted. Neither are rows gone from a table whose newest row is still there, since that is a purge of old rows. A drop of fewer than 100 rows is a warning, and so is a drop in a file tree or mailbox that held fewer than 50 items. | | table_drop | Critical | A table or collection from the snapshot before is missing. It is not flagged when a newer partition of the same name replaced it (`messages_2026_10_15` gone, `messages_2026_10_18` new). It is also not flagged when a new table has the same columns, which is a rename. A missing table that held no rows is a warning. | | empty_snapshot | Critical | The snapshot has no tables, files or messages where the one before had some. A file tree or mailbox that held fewer than 50 items is a warning. | | extension_mismatch | Warning | A Postgres extension from the snapshot before is missing. Without it, the schema can fail to restore. | | stale_table | Warning | A table gained at least 5 rows and 1%, but its newest `updated_at`, `modified_at` or `created_at` did not move. That is what rows loaded from an old export look like. Check that the surface's connection string points at the live database. Fewer rows with the same newest row is a purge of old rows, and is not flagged. This warning does not make SafeGrd keep the snapshot before any longer. | | column_drop | Warning | A table from the snapshot before has lost a column, by name. The table and its rows are still there, but the data in that column is gone, and no row count would show it. If a column of the same type was added at the same time, it is taken as a rename and not flagged. A migration that drops a column on purpose looks the same, so this is only a warning, and the snapshot before is not kept any longer. Once you have checked the change, resolve it, and the next backup is compared with this one. Columns are compared only when both backups recorded them. | A warning is sent once. The next backup is compared with the flagged snapshot, so the same change does not raise it again. A critical anomaly works differently. The next backups are compared with the snapshot before the flagged one, until you resolve the anomaly. After that, the next backup is compared with the flagged snapshot. ### What each snapshot records Each completed snapshot carries Threat Shield's record. It names the snapshot it was compared with, and lists each check with the numbers it used and whether it passed. It also lists what changed between the two: tables added and gone, tables whose definition changed, columns lost or gained, and the tables whose row count moved most. The console shows this record on every snapshot, including the ones that were not flagged. A surface's first backup records that there was nothing to compare it with. Each PostgreSQL, MySQL, MariaDB and SQLite backup also records every table's columns (name, type, not null, primary key) and a hash of the table's definition. A MongoDB backup records each collection the same way. It lists the top-level fields across every document in the backup, with their BSON type (`mixed` where documents disagree). `_id` counts as the primary key, and a field that every document has counts as not null. If a field disappears from every document, you get the same `column_drop` warning. Column names are schema, like the table names already recorded. The rows stay sealed in the backup. On the surface's _Tables_ tab in the console, you can see each table across the last 30 backups: its row count as a line, its size, its newest row, when its definition changed, and what the last drill found for it. Click a row to see its columns ([details](https://safegrd.dev/docs/surfaces/databases#tables)). ### Newest row and sample hash For each table, a PostgreSQL backup records two more things, and the console shows both in a snapshot's table list: - The newest value in the first timestamp column the table has, out of `updated_at`, `modified_at` and `created_at`. It is read under the backup's own snapshot. A table with none of the three has no newest row. On a table over 256 MB, the value is read only when an index leads with that column, so the backup does not read the table twice. - A SHA-256 of the table's first 100 rows by primary key. A table without a primary key has no hash. SafeGrd records the hash but does not alert on it, because any update to those rows changes it. ## Why the good snapshot is still there The alert names the last known-good snapshot, and pruning keeps it while you look into the anomaly ([Storage and retention](https://safegrd.dev/docs/storage)). Under Object Lock, the snapshots taken before the damage cannot be deleted until their locks end anyway. Threat Shield does not freeze or extend any lock. A lock is set once, when the snapshot is written. To keep a known-good snapshot longer, restore it and back it up again, or copy it out with `safegrd export`. ## Alerts Threat Shield sends each anomaly as the `anomaly` event ([Alerts](https://safegrd.dev/docs/alerts)). You see it in three places: - Your organization's webhook (Slack, Discord or generic JSON) gets the surface, what dropped, and the known-good snapshot. - On paid plans, the organization's owners and admins get an email, once an owner or admin turns email alerts on under _Settings → Alerts_. - The console marks the surface and the snapshot as anomalous. ## When you drop data on purpose If you prune an audit log or drop a table on purpose, the next backup flags the drop. For a warning you do not need to do anything, because the next backup is compared with the flagged one anyway. For a critical anomaly, resolve it in the console. The next backup is then compared with the flagged snapshot. --- Source: https://safegrd.dev/docs/attestation # The attestation record SafeGrd keeps an append-only record of when each snapshot was created, when it was tested, and what was restored. You can verify the chain offline with SafeGrd's public key. ## How the chain works Each Fire Drill produces a certificate, and every certificate carries the hash of the one before it. The first record has no certificate before it, so its `PrevHash` is the literal string `genesis`. SafeGrd's attestation key signs each certificate with Ed25519. Edit an older certificate and every hash after it stops matching. Remove one and the next record links to nothing. Change a date and the signature no longer verifies. For a database, the drill record also lists each table whose row count changed since the previous backup. The list is `table_changes`, and `table_changes_from` names the backup it was compared with (the one Threat Shield compared it with). The list also names the tables added and gone, and counts the unchanged ones. The list stops at 200 tables, largest change first, and `table_changes_omitted` counts the rest. A partition rotated out for a newer one of the same name shows as one table gone and one added. The signature covers a SHA-256 digest of the list, so a changed count fails the check. A record from before 30 October 2026 has no list and verifies as it always did. To verify a record that has one, use `safegrd verify history` from a CLI released after that date. ## Verify it yourself ``` # check the full chain for a host against SafeGrd safegrd verify history --node node-prod-db-primary # machine-readable, for a compliance pipeline safegrd verify history --node node-prod-db-primary --json ``` The command checks that: - the first record says `genesis` - every record links to the certificate hash of the one before it - every Ed25519 signature is valid under the key the record names - no record was changed, removed or inserted ### When the signing key changes Each record names the key that signed it (`signing_key_id`). When SafeGrd rotates its attestation key, the old key's public half stays published at `/api/v1/attestations/public-key` under `retired`, with the time it stopped signing. The records it signed keep verifying. `safegrd verify history` checks each record under its own key. It prints which keys signed the chain and how many records each signed. It refuses a record signed with a retired key if the record is dated after the key retired, or if it follows a record signed with the newer key. ### Offline and air-gapped audit ``` # online, on an enrolled host: check the chain and save it, with the public key safegrd verify history --save ./exported-chain.json curl -s https://safegrd.dev/api/v1/attestations/public-key | jq -r .public_key > ./attestation.pub # later, with no network at all safegrd verify history --file ./exported-chain.json --key file:./attestation.pub ``` Give an auditor the exported JSON and the public key, and they can verify the chain on a machine with no network. `--key` also takes the hex key inline. Either way, every record is checked against that one key. For a chain that spans a key rotation, save the published key set while you are online and pass that instead: ``` # the active key and every retired one, as one file curl -s https://safegrd.dev/api/v1/attestations/public-key > keys.json safegrd verify history --file ./exported-chain.json --keys-file ./keys.json # or without the CLI: paste both files into /tools/attestation-verifier ``` An offline check cannot see records the file leaves out. For a chain exported before a key rotation, also run `safegrd verify history --node` against the server, which walks every record the server holds, and compare the two. You check an Evidence Pack the same way, using the file as it was downloaded. Run `safegrd verify history --file evidence-pack.json`, or paste it into the verifier page. The pack holds every host's records, and each host's chain is checked on its own. A pack for one quarter starts each chain partway through. For each host, it names the record just before the quarter in `chain_anchors`, and that host's first record in the pack has to link to it. The pack carries the keys that signed it. The check uses those unless you pass `--key` or `--keys-file` with keys you saved yourself. The console can also download the pack as a PDF, laid out for an auditor who reads on paper. The PDF is written from the JSON and has no signature of its own. Give the auditor both files and run the check on the JSON. ## Sample records You can download and check these files. They come from a demo organization, where the daemon backed up a PostgreSQL database and a directory into a bucket under S3 Object Lock in compliance mode. It then restored the database into an empty one and compared it with the backup. | File | What it is | | --- | --- | | [certificate.json](https://safegrd.dev/static/samples/certificate.json) | One drill’s signed record: what was restored, every check and its result, and the hash of the record before it. | | [drill-chain.json](https://safegrd.dev/static/samples/drill-chain.json) | Every drill record for that database, oldest first, as the chain the command below checks. | | [attestation-key.txt](https://safegrd.dev/static/samples/attestation-key.txt) | The Ed25519 public key of the demo server, which signed them, hex-encoded. It is not safegrd.dev’s key, so check these files against this one. | | [evidence-pack.json](https://safegrd.dev/static/samples/evidence-pack.json) | The Evidence Pack a Scale organization downloads: drill records, snapshot retention and the controls they are evidence for. It is not a certification or an audit opinion. | | [evidence-pack.pdf](https://safegrd.dev/static/samples/evidence-pack.pdf) | The same pack as the PDF the console downloads, for an auditor who reads on paper. The signatures are over the JSON records, so a check runs on the JSON file. | ``` # check the sample chain offline safegrd verify history --file drill-chain.json --key file:attestation-key.txt # and the sample Evidence Pack, two hosts' chains safegrd verify history --file evidence-pack.json --key file:attestation-key.txt ``` Change any field in `drill-chain.json` and the same command reports the record whose signature no longer matches. ## Public certificates An owner or admin can publish a passed drill as a page that anyone with the link can open. In the console, open the Fire Drills tab, choose **History**, then **Certificate** on the drill, and **Publish certificate**. The link looks like `https://safegrd.dev/c/sgc_…`. The address is random, so nobody finds a certificate unless you give them the link, and search engines are told not to list it. **Withdraw** turns the link off within a minute. If you retire the surface, every certificate published from it is turned off. The page shows: - the organization’s name - the kind of surface (a PostgreSQL database, a file tree, a mailbox) - how the drill ran: restored into a sandbox, restored in full, or decrypted and read in memory - when the backup was taken and when the drill passed - who made the backup, if another tool made it and SafeGrd checked it ([watched backups](https://safegrd.dev/docs/surfaces/postgres#watch)). The page names the tool, such as Coolify. It says the drill proves the backup restores, and does not claim the backup holds the whole database. It never shows where the backup is kept. - where the drill ran: on a machine SafeGrd started for it, or on one of the organization’s own hosts, which reported it to SafeGrd (the host is not named) - the certificate’s id and the key that signed it By default it shows nothing else: no table or file names, row or file counts, sizes, restore times, surface names, host names, snapshot ids or storage. An owner or admin can add the surface’s name, the restore time and what was restored (tables and rows, files and directories, or emails and folders). Click **Change** under **Badge and public page** on the Fire Drills tab. The choice applies to every public certificate the organization publishes. Host names, snapshot ids and storage are never shown. Members of the organization see the whole record in the console. ### A surface’s page, for a README badge A drill’s certificate covers one point in time. For a link that stays current, publish the surface’s page from **Badge and public page** on the Fire Drills tab. Its address, `https://safegrd.dev/c/sgs_…`, shows the surface’s newest passed drill. While the page is published, every drill that passes is published with it. The badge markdown the console gives you links to this page. If a newer drill did not pass, the page says so and shows no seal. When you withdraw the page, the drill certificates it published are withdrawn too. Drills you published one by one stay published. ### A project’s page To show every surface in a project on one page, publish it from **Project pages** on the same card. Its address is `https://safegrd.dev/c/sgp_…`. For each surface it lists the kind, the last backup and the newest drill, read when the page is opened and signed the same way. Where a surface has its own page, the project page links to it. The seal shows when every surface’s newest drill passed. Unlike a surface’s page, the project page says how many surfaces the project has and what kind they are. Surface names appear only if you chose to show them under **What certificates show**. Withdrawing the project page leaves surface pages and drill certificates as they were. The page checks the certificate’s Ed25519 signature in the browser before it shows the seal. To check it yourself, without the page, use OpenSSL 3: ``` # the signed statement, and its signature curl -s https://safegrd.dev/api/v1/certificates/sgc_… > cert.json # or sgs_…, sgp_… jq -j .statement cert.json > statement.json jq -r .signature cert.json | xxd -r -p > statement.sig # the attestation public key, as PEM curl -s https://safegrd.dev/api/v1/attestations/public-key | jq -r .public_key \ | sed 's/^/302a300506032b6570032100/' | xxd -r -p | openssl pkey -pubin -inform DER -out safegrd.pem # prints "Signature Verified Successfully" openssl pkeyutl -verify -pubin -inkey safegrd.pem -rawin -in statement.json -sigfile statement.sig ``` The statement’s `record` field ties it to the full drill record without showing the record. It is the SHA-256 of a random salt, a `|`, and the record’s signed fields. The console shows the salt and the command that recomputes the hash, so you can show an auditor which record a certificate stands for. ## When a report does not reach us A backup still completes if its report does not reach SafeGrd. The CLI prints a warning to stderr, and the backup counts as done on the host. The run is marked attested only after SafeGrd acknowledges its report, so the host and the attestation record agree on which runs were recorded. --- Source: https://safegrd.dev/docs/alerts # Alerts SafeGrd tells you when a backup or a drill needs your attention. Alerts come from SafeGrd's server rather than your hosts, so you still hear about a host that has gone offline. ## Where alerts go | Channel | Plans | How | | --- | --- | --- | | The console | Every plan | Nothing to set up. _Alerts_ in the console's top bar lists every alert from the last 30 days, newest first, with the surface it is about. Every member of the organization sees it, viewers included. An access token can read it from `GET /api/v1/orgs/{id}/events`. | | Webhook | Every plan, Free included | Paste a URL under _Settings → Alerts_ in the console, then press _Send a test_. Slack incoming-webhook URLs get Slack's message format, Discord webhook URLs get Discord's, and Teams workflow URLs get an Adaptive Card. Any other URL gets the JSON below. [Getting the URL](https://safegrd.dev/docs/alerts#webhook-url) says where Slack, Discord and Teams show it. | | PagerDuty | Every plan, Free included | Paste an Events API v2 integration key under _Settings → Alerts_. A failure or a warning opens an incident on that service, and its recovery resolves it. You get one incident per surface and condition. A notice (storage at 80%, a bucket caught up) opens no incident. [Getting the key](https://safegrd.dev/docs/alerts#pagerduty) says where PagerDuty shows it. [What fires](https://safegrd.dev/docs/alerts#events) lists which alerts resolve on their own. | | Email | Paid plans, off until you turn it on | An owner or admin turns on _Email alerts to owners and admins_ under _Settings → Alerts_. Alerts then go to every owner and admin of the organization, at the address they sign in with. Email is there to send you to the console, so it is capped: at most one email per kind of alert for a surface in a day, and one per kind across the organization in an hour. The webhook and the console get every event. | Hosts do not send alerts, so a host's `config.yaml` has no alert settings. If `safegrd doctor` finds an `alert:` block, it says so. The webhook must be a public address. SafeGrd refuses a URL that points at a private or loopback network, both when you save it and again when it connects. It does not follow redirects. ## Getting the URL ### Discord - In the Discord server, open _Server Settings → Integrations → Webhooks_. You need the Manage Webhooks permission on the server. - Press _New Webhook_, name it (for example SafeGrd) and choose the channel alerts go to. - Press _Copy Webhook URL_. It looks like `https://discord.com/api/webhooks/1234567890/AbC...`. - In the SafeGrd console, paste it under _Settings → Alerts_, press _Save_, then _Send a test_. The test message shows up in that channel. ### Slack - Go to [api.slack.com/apps](https://api.slack.com/apps), press _Create New App_, choose _From scratch_ and pick your workspace. - Under _Incoming Webhooks_, turn on _Activate Incoming Webhooks_. - Press _Add New Webhook to Workspace_ and choose the channel alerts go to. If your workspace limits app installs, an admin has to approve it first. - Copy the URL Slack shows. It starts with `https://hooks.slack.com/services/`. - Paste it under _Settings → Alerts_, press _Save_, then _Send a test_. ### Microsoft Teams - In Teams, open the channel, press _More options (…)_ and choose _Workflows_. - Pick the template _Send webhook alerts to a channel_ (its trigger is _When a Teams webhook request is received_), choose the team and channel, and press _Add workflow_. - Copy the URL the workflow shows. It is on `logic.azure.com`, `powerplatform.com` or `powerautomate.com`, with a `sig=` parameter. - Paste it under _Settings → Alerts_, press _Save_, then _Send a test_. The test shows up in that channel as a card. Microsoft retired the Office 365 connector webhooks (`webhook.office.com`) in May 2026. If one of those URLs is still pasted, it gets the same card. A workflow belongs to the person who made it, so add a co-owner in Power Automate to keep it running after they leave. Paste the URL exactly as Slack, Discord or Teams gives it. SafeGrd picks the message format from the address: `hooks.slack.com` for Slack, `discord.com/api/webhooks/` for Discord, and `logic.azure.com`, `powerplatform.com` or `powerautomate.com` for Teams. A URL behind a shortener or proxy gets the generic JSON, which Slack, Discord and Teams reject. Anyone with the URL can post to that channel, so keep it out of tickets and chat. To replace it, delete the webhook in Slack, Discord or Teams, make a new one and paste that. ### PagerDuty - In PagerDuty, open the service alerts should page for and its _Integrations_ tab. - Press _Add an integration_, choose _Events API V2_ and press _Add_. - Copy the _Integration Key_: 32 letters and digits. - Paste it under _Settings → Alerts_, press _Save key_, then _Send a test_. The test opens an incident on the service and resolves it at once, so it shows on the service's timeline and pages nobody for long. Each incident's deduplication key names the organization, the surface and the condition. If a surface fails again while its incident is open, the alert adds to that incident instead of opening a second one, and the recovery alert resolves the right one. The severity is _error_ for a failure and _warning_ for something late or degraded. Your service's urgency rules decide who is paged. ## What fires Each condition sends one alert when it starts and one when it clears, so a surface that stays broken does not flood the channel with an alert per check. These alerts clear each other: - `backup_failed`, cleared by `backup_recovered` - `silent` and `overdue`, cleared by `recovered` - `unreachable`, cleared by `reachable_again` - `copy_behind`, cleared by `copy_caught_up` The clearing alert carries `clears`, naming the condition it ends, and resolves its PagerDuty incident. `drill_failed`, `drill_blocked`, `anomaly`, `not_started`, `clock_skew`, `duplicate_host` and `hosted_storage_full` have no clearing alert. Resolve those incidents yourself once the cause is fixed. Do the same for an incident on a surface you remove, or add again under a new host. Nothing resolves it, because the clearing alert is keyed to the surface as it was. | Event | When | | --- | --- | | backup_failed | A surface starts failing to back up. You get one alert when it starts, and another only if it fails with a different error. The daemon tries 3 times, 5 minutes apart, then waits for the next scheduled backup. | | backup_recovered | That surface backs up again. | | drill_failed | A Fire Drill ran and the snapshot did not restore whole. Also sent when the drill stopped before restoring, because the host could not open storage, `drill.sandbox_url` was refused, or the daemon stopped during the drill. | | drill_blocked | A Fire Drill is due and the host cannot run it, because it does not have the private key or lacks the disk space to restore the snapshot. Sent to the webhook and shown in the console, but not emailed. | | silent | A daemon has missed three check-ins, and at least 15 minutes have passed. Nothing is backing up its surfaces. | | not_started | A host enrolled more than a day ago has backed up nothing since. No surface has registered from it and no daemon has reported from it. The alert says when it enrolled and what to run on it. A host enrolled only to restore gets it once, and needs no further setup. No alert is sent when the host starts. Its first backup shows in the console. | | overdue | A surface has gone twice its schedule without a backup, whether or not its host is checking in. For a surface that checks another tool’s backups ([watch:](https://safegrd.dev/docs/surfaces/postgres#watch)), that tool has written no new backup in twice the schedule. | | unreachable | On three ticks in a row, the daemon could not open a surface’s database, reach its sink, or move its erasure record to the sink ([checks between backups](https://safegrd.dev/docs/daemon#probe)). The alert gives the error and since when. The next backup fails unless the surface answers by then. | | reachable_again | The daemon opened that surface’s database and reached its sink again. | | copy_behind | One of a surface's other buckets has been missing backups for a day ([A second bucket](https://safegrd.dev/docs/storage#second-bucket)). The alert names the bucket, how many backups it is missing and since when. | | copy_caught_up | That bucket holds every backup again. | | clock_skew | A host's clock is more than 5 minutes off from SafeGrd's server. The host's clock sets snapshot dates, which backup is kept longest, and Object Lock dates in your own bucket. Sent once a day per host. | | duplicate_host | Two machines are taking turns reporting as one host, which happens when a config is copied to a second machine. The alert names each machine by hostname. Run `safegrd enroll` on the copy to give it a host of its own. Sent once a day per host. | | anomaly | Threat Shield flagged a snapshot: a drop in rows (>20%), files (25%) or messages (30%), a dropped table, an empty snapshot, or a missing extension. The alert names the last known-good snapshot to restore from ([Threat Shield](https://safegrd.dev/docs/threat-shield)). | | kept_undrilled | A project that locks only its kept copies kept a weekly or monthly copy with no passing Fire Drill, because no backup in that week or month was drilled in time. The copy is locked. Drill it to prove it restores ([kept copies](https://safegrd.dev/docs/storage#kept-copies)). | | drill_read_unnamed | A Fire Drill read objects the backup did not record as part of its snapshot. The drill passed, but the snapshot is not kept as a weekly or monthly copy until a drill reads only recorded objects. Upgrade safegrd on the host that took it. | | hosted_storage_80 | Hosted storage is 80% full of locked backups. On a paid plan backups continue past 100% and the excess is billed per GB-month. On Free or a trial, new backups are refused at 100%. | | hosted_storage_over | A paid plan’s hosted storage passed its included storage. Backups continue, and the average held above it each billing period is billed per GB-month ([pricing](https://safegrd.dev/pricing)). | | hosted_storage_full | Hosted storage refused a backup. This happens on Free or a trial at its included storage, on a lapsed plan past its margin, or when an organization reaches its hosted storage limit. New backups are refused until locks expire or the organization moves to a larger plan or its own bucket. Existing backups stay restorable. | | recovered | A silent host is checking in again, or an overdue surface is backing up again. | | test | You pressed _Send a test_. | Surfaces in an archived project do not alert. ## What a webhook receives Any URL other than Slack, Discord or Teams receives one JSON object per alert, sent with `Content-Type: application/json` and `User-Agent: SafeGrd-Alerts/1`: ``` { "id": "alrt-Xq3v9KpL2mZa", "event": "overdue", "title": "Backup overdue", "text": "No backup for twice this surface's schedule (@daily). Last backup: 2026-09-21 03:00 UTC.", "org_id": "org-2a5755f2", "node_id": "node-53274c18--docs", "node_name": "docs", "at": "2026-09-23T03:00:00Z", "severity": "warning" } ``` `severity` is `error`, `warning` or `info`. An alert that ends a condition carries `"clears"` with that condition's name. For example, a `recovered` after an `overdue` has `"clears": "overdue"`. Slack receives `{"text": "*title*\ntext"}`, Discord `{"content": "**title**\ntext"}` and Teams a message with one Adaptive Card holding the title and the text. Alerts contain operational metadata and never include backup contents. If a send fails, SafeGrd tries again after 1, 5 and 15 minutes, then after 1, 3 and 6 hours. It does not retry a 4xx answer other than 408 or 429. An alert can arrive more than once, for example when SafeGrd restarts during a send. `id` is the same on every attempt, so a receiver that must act only once can skip an `id` it has already seen. --- Source: https://safegrd.dev/docs/mcp # AI agents (MCP) Your coding agent can check your backups over the Model Context Protocol (MCP), take a backup before a risky migration and run a drill to show it restores. Nobody has to open the console. SafeGrd has two MCP servers. The remote one reads every host's state and asks for a backup or a drill. It never holds data or decrypts anything. The local one runs on a host and backs up, verifies and restores with that host's config and key. Both servers come with every plan, including the free plan. A tool can ask only for what your plan allows. On the free plan, a Fire Drill runs in memory, once a week. | Server | Where it runs | What it can do | | --- | --- | --- | | Remote | `https://safegrd.dev/mcp` | Reads organizations, projects, surfaces, snapshots and drills. Asks for a backup or a Fire Drill now. Reads storage use and drill statistics. Gives you the link or the commands that add a surface or a host. | | Local | `safegrd mcp`, on the host | Status, list, doctor, back up, verify, restore into a new or empty target, and export. | ## What an agent cannot do No tool deletes a snapshot, changes where backups are stored, rotates a key, or changes billing or members. The server refuses these to a personal access token on every API route, so a token cannot do them through REST either. Nobody can delete a snapshot under compliance-mode Object Lock before its lock expires. Every tool is marked non-destructive in its MCP annotations, and the read tools are marked read-only, so a client can run them without asking you each time. To take a snapshot before an agent runs a command that could damage the database, use the [Agent guard](https://safegrd.dev/docs/agents). The console's _Tokens_ page lists every _request_backup_ and _request_drill_, along with the same requests made from the console. Each row names the person, the token's name, whether the request came over MCP, and whether it was accepted. A sign-in from claude.ai or ChatGPT is a personal access token with the client's name on it, so the same refusals apply to it. ## Add a database from a chat Ask the agent to back up a database SafeGrd does not back up yet. It gives you a console link that opens _Back up on SafeGrd_ on the right project. For Supabase, DigitalOcean and Heroku, you sign in to the platform on that page and pick the database, so you type no password. For any other database, paste its connection string on that page. Do not paste it into the chat. When you choose _Add_, SafeGrd takes the first backup and runs a Fire Drill on it, and the agent can see the new surface. For a database on a machine you run, the agent gives you the commands instead: the installer, `safegrd login`, `safegrd enroll` and the first backup. An agent with a shell on that machine, such as Claude Code, can run them itself. You approve the sign-in in a browser. ## The remote server ### Sign in from Claude or ChatGPT Add `https://safegrd.dev/mcp` to your client as a connector. The client opens SafeGrd, you sign in and approve the connection, and the agent can use SafeGrd. There is no token to copy. **claude.ai and Claude Desktop** - Open [Customize, then Connectors](https://claude.ai/customize/connectors), and choose _Add_, then _Add custom connector_. - Enter `SafeGrd` as the name and `https://safegrd.dev/mcp` as the URL. Choose _Continue_. - Keep the sign-in settings Claude finds, choose _Sign in now_, then _Add_. - Sign in to SafeGrd on the page that opens and approve the connection, as described [below](https://safegrd.dev/docs/mcp#approve). - In a chat, choose _+_, then _Connectors_, and turn on SafeGrd. On a Team or Enterprise plan, an owner adds the connector once under _Organization settings_, _Connectors_, with the same name and URL. Each member then chooses _Connect_ next to SafeGrd under _Customize_, _Connectors_, and signs in to SafeGrd as themselves. A connector you add on claude.ai also shows up in Claude Desktop. **ChatGPT** - Open [chatgpt.com/plugins](https://chatgpt.com/plugins), choose _+_, then _Add custom MCP server_. - Enter `SafeGrd` as the name. Under _Connection_, choose the public endpoint and enter `https://safegrd.dev/mcp`. - Set the authentication to OAuth, accept the warning about custom servers, and choose _Create as a plugin_. - Install the plugin. When ChatGPT opens SafeGrd, sign in and approve the connection. - In a new chat, type `@` and choose SafeGrd. **Claude Code** - Add the server with no header: ``` claude mcp add --transport http safegrd https://safegrd.dev/mcp ``` - Restart Claude Code. A session that was already open lists the server only after a restart (`claude --continue` keeps the conversation). - In Claude Code, run `/mcp`, choose _safegrd_, then _Authenticate_. Your browser opens SafeGrd. Sign in and approve the connection. You can install the Claude Code plugin instead. It adds the same server, signed in the same way, plus skills that back up and run a Fire Drill before a risky change and report on backup health. After installing, run `/mcp` and choose _Authenticate_ as above. The source is at [safegrd/agent-plugins](https://github.com/safegrd/agent-plugins). ``` /plugin marketplace add safegrd/agent-plugins /plugin install safegrd@safegrd ``` When you sign in, SafeGrd's _Connect an agent_ page shows the name the client gave itself and the address it will send you back to. If that is not the app you are connecting, choose _Deny_. Otherwise, pick the organization the agent may reach, limit it to one project if you want, and choose _Approve_. Approving creates an access token, named after the client, under _Tokens_ in the console. Revoke it there and the agent loses access on its next call. The client's token lasts an hour and renews itself. It works on the MCP server only, and the REST API refuses it. The sign-in is OAuth 2.1 with PKCE. A client reads the server's metadata at `/.well-known/oauth-authorization-server`. It then either registers itself or SafeGrd reads its client metadata document. ### Or a personal access token For a client that cannot sign in, such as CI, a script or Cursor, create a personal access token in the console under _Tokens_ and send it as a bearer header. In Claude Code: ``` claude mcp add --transport http safegrd https://safegrd.dev/mcp \ --header "Authorization: Bearer sg_pat_..." ``` Any client that supports the Streamable HTTP transport and a bearer header works the same way. The server keeps no session and checks every call against the token as it is now. If you revoke the token in the console, the agent stops on its next call. ### Other clients In Cursor, add this to `~/.cursor/mcp.json`, or to `.cursor/mcp.json` in a project: ``` { "mcpServers": { "safegrd": { "url": "https://safegrd.dev/mcp", "headers": { "Authorization": "Bearer sg_pat_..." } } } } ``` In VS Code, add this to `.vscode/mcp.json`. VS Code asks for the token the first time it starts the server and stores it, so the file holds no secret: ``` { "inputs": [ { "type": "promptString", "id": "safegrd-token", "description": "SafeGrd personal access token", "password": true } ], "servers": { "safegrd": { "type": "http", "url": "https://safegrd.dev/mcp", "headers": { "Authorization": "Bearer ${input:safegrd-token}" } } } } ``` Clients that browse the [MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=dev.safegrd) can install SafeGrd from there, as `dev.safegrd/safegrd`. It is also on [Smithery](https://smithery.ai/servers/safegrd/safegrd) as `safegrd/safegrd`. On Smithery, enter the whole header value, `Bearer sg_pat_...`, in the `x-safegrd-token` field. | Tool | Arguments | What it does | | --- | --- | --- | | list_organizations | | Organizations the token belongs to, with plan and quotas. | | list_projects | org_id | An organization's projects, such as production and staging. | | list_surfaces | project_id, limit, cursor (all optional) | Databases, file trees, mailboxes and hosts, with last backup, drill and alert state. | | get_surface | node_id | One surface or host. | | list_snapshots | node_id; limit, cursor (optional) | A surface's snapshots, newest first: when, how big, locked until when, and whether a drill verified them. | | get_snapshot | snapshot_id | One snapshot's recorded metadata: digests, counts, lock, Threat Shield verdict. | | list_drills | node_id; limit, cursor (optional) | A surface's Fire Drill reports, newest first, with their assertions and signatures. | | request_backup | node_id | Ask for a backup now. Refused within an hour of the last backup, because each one is locked and stored until it expires. | | request_drill | node_id | Ask for a Fire Drill of the latest snapshot now. Counted against the plan's drill cadence. | | get_storage_usage | org_id | Hosted storage: whether it is offered, the quota and how much is locked. | | get_drill_stats | org_id | Fire Drill pass rate and results across an organization. | The remote server does not run backups or drills itself. On a surface your host backs up, the host's [daemon](https://safegrd.dev/docs/daemon) picks up the request at its next check-in. So _request_backup_ and _request_drill_ do nothing unless the daemon is running on that host. A surface set to _Back up on SafeGrd_ or _Drill on SafeGrd_ in the console is queued on SafeGrd instead, and needs no daemon. If the API refuses a request, the tool error carries the API's own words, so the agent can tell you why. The three list tools return `{"items": [...], "next_cursor": "..."}`, 25 items at a time unless you set `limit` (up to 200). For the next page, pass `next_cursor` back as `cursor`. The last page has no `next_cursor`. ## The local server An agent running on the host itself can use `safegrd mcp`, which serves that host's SafeGrd over stdio. In Claude Code: ``` claude mcp add safegrd -- safegrd mcp ``` If the host runs the CLI as a container, run the same image with the volume you enrolled with, which holds its config and key: ``` claude mcp add safegrd -- docker run -i --rm \ -v safegrd:/home/safegrd/.safegrd ghcr.io/safegrd/cli mcp ``` Each tool runs the CLI command of the same name with this host's config, and refuses whatever that command refuses. Two more limits apply: - No tool takes a private key. The tools use the host's `key_path`, so your key never passes through an agent's context. - _restore_ writes only into a new or empty directory, or into a database it has checked is empty. An agent cannot restore over data. | Tool | Arguments | Runs | | --- | --- | --- | | list | | `safegrd list` | | doctor | | `safegrd doctor` | | backup | surface or files (optional) | `safegrd backup --surface` for one surface of the host's config, `--files` for a directory tree, or the configured database | | verify | snapshot_id, sandbox_url (optional) | `safegrd verify --in-memory`, or a full drill into an empty database | | restore | snapshot_id, and target_dir or target_url | `safegrd restore` into an empty target | | export | to_dir, or to_bucket and to_endpoint | `safegrd export`, still encrypted | ## The REST API underneath Every remote tool calls the same `/api/v1` routes the console uses. [/openapi.json](https://safegrd.dev/openapi.json) describes the routes an agent needs, and [/llms.txt](https://safegrd.dev/llms.txt) points an agent at these docs. You can see the remote server's tools without a token at `/.well-known/mcp/server-card.json`. --- Source: https://safegrd.dev/docs/api # API reference The part of SafeGrd's /api/v1 an agent needs: read backups and Fire Drills, and ask for a backup or a drill. Authenticate with a personal access token from the console (Tokens). A personal access token cannot delete, disable, re-route or re-key anything, on any route. The same operations are served as MCP tools at https://safegrd.dev/mcp; see https://safegrd.dev/docs/mcp. Make a personal access token under _Tokens_ in the console, then send requests to `https://safegrd.dev` with `Authorization: Bearer sg_pat_…`. This page is rendered from [/openapi.json](https://safegrd.dev/openapi.json), the document an agent reads, so the two cannot disagree. Every other `/api/v1` route (sign-in, billing, members, sinks, keys, host enrollment) is internal. Those routes change between releases, and they refuse a personal access token. ## Operations ### The plan catalogue: prices, quotas, drill cadence, hosted storage GET /api/v1/plans (no token needed) | Response | Meaning | Body | | --- | --- | --- | | 200 | Every plan | array | ### What a backup host's key needs in your own bucket, per provider: steps, least-privilege policy, and for AWS a CloudFormation and a Terraform template that create the bucket with Object Lock GET /api/v1/storage/bucket-setup (no token needed) | Parameter | In | Meaning | | --- | --- | --- | | provider (required) | query | string | | bucket (required) | query | string | | expiry | query | 1 to add what opt-in expiry in your own bucket needs | | Response | Meaning | Body | | --- | --- | --- | | 200 | The setup | object | | 400 | An unknown provider or a bucket name S3 does not allow | | ### Organizations the caller belongs to, with plan and quotas GET /api/v1/orgs | Response | Meaning | Body | | --- | --- | --- | | 200 | Organizations | array | | 401 | Refused. The body says why. | Error | ### The commands that enroll a host and take its first backup. Changes nothing. GET /api/v1/orgs/{org_id}/add-host | Parameter | In | Meaning | | --- | --- | --- | | org_id (required) | path | string | | project_id | query | Project to add it to. Default: the organization's default project. | | Response | Meaning | Body | | --- | --- | --- | | 200 | The commands and steps | AddGuide | | 400 | Refused. The body says why. | Error | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### The console link and steps to add a surface. Changes nothing. GET /api/v1/orgs/{org_id}/add-surface | Parameter | In | Meaning | | --- | --- | --- | | org_id (required) | path | string | | kind (required) | query | Where the data is. | | project_id | query | Project to add it to. Default: the organization's default project. | | Response | Meaning | Body | | --- | --- | --- | | 200 | The link, the steps, and whether this caller can finish them | AddGuide | | 400 | Refused. The body says why. | Error | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### Fire Drill statistics across an organization GET /api/v1/orgs/{org_id}/drill-stats | Parameter | In | Meaning | | --- | --- | --- | | org_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | Statistics | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### An organization's projects GET /api/v1/orgs/{org_id}/projects | Parameter | In | Meaning | | --- | --- | --- | | org_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | Projects | array | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### Each surface's recovery point and restore time, measured from its snapshots and drills GET /api/v1/orgs/{org_id}/recovery | Parameter | In | Meaning | | --- | --- | --- | | org_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | One row per surface, with the window and sample counts behind each figure | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### Hosted storage: whether it is offered, the quota, and how much is locked GET /api/v1/orgs/{org_id}/storage | Parameter | In | Meaning | | --- | --- | --- | | org_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | Usage | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### Protected surfaces and hosts, with last backup, drill and alert state. Each carries last_snapshot, its newest backup's summary GET /api/v1/nodes | Parameter | In | Meaning | | --- | --- | --- | | org_id | query | Only this organization, which must be one of the caller's | | project_id | query | Only this project's surfaces | | limit | query | Ask for a page of this many, newest first (default 50, at most 200). With limit or cursor the response is a Page; without either it is the whole list as a bare array. | | cursor | query | next_cursor from the previous page. Opaque; a key, not an offset, so rows arriving meanwhile never repeat or skip one. | | Response | Meaning | Body | | --- | --- | --- | | 200 | Surfaces | | | 400 | Refused. The body says why. | Error | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | ### One surface or host GET /api/v1/nodes/{node_id} | Parameter | In | Meaning | | --- | --- | --- | | node_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The surface | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### Ask for a backup of the surface now POST /api/v1/nodes/{node_id}/backup-now The host's daemon runs it at its next check-in. A surface SafeGrd backs up is queued on SafeGrd instead (202 with a job_id). Every backup is locked and cannot be deleted before its retention expires, so a request within an hour of the last backup is refused with 429. A host that does not run the daemon cannot pick a request up (409). | Parameter | In | Meaning | | --- | --- | --- | | node_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | Already requested | object | | 202 | Requested | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | | 409 | Refused. The body says why. | Error | | 429 | Refused. The body says why. | Error | ### Read what each of a surface's other buckets holds GET /api/v1/nodes/{node_id}/copies Per bucket: backups held, any not written yet, the latest lock, how many it lacks and since when, as the host last said, and when a drill last restored from it. | Parameter | In | Meaning | | --- | --- | --- | | node_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The surface's buckets, under buckets | object | | 403 | Refused. The body says why. | Error | ### Ask for a Fire Drill of the surface's latest snapshot now POST /api/v1/nodes/{node_id}/drill-now The host's daemon runs it at its next check-in. A surface set to drill on SafeGrd is queued there instead (202 with a job_id) and needs no daemon. Counted against the plan's drill cadence: refused with 429 until the next drill is due. Refused with 409 when nothing would run it: no daemon on the host, or no backup yet. | Parameter | In | Meaning | | --- | --- | --- | | node_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | Already requested | object | | 202 | Requested | object | | 401 | Refused. The body says why. | Error | | 402 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | | 409 | Refused. The body says why. | Error | | 429 | Refused. The body says why. | Error | ### List a surface's restores asked for in the console GET /api/v1/nodes/{node_id}/restores Newest first, with who asked, the snapshot, the target without its secret, what was restored, the state and the counts. A completed restore carries its signed record under certificate. Asking for a restore needs a console session of an owner or an admin, so an access token cannot. | Parameter | In | Meaning | | --- | --- | --- | | node_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The surface's restores, under restores | object | | 403 | Refused. The body says why. | Error | ### A surface's snapshots: when, how big, locked until when, whether a drill verified them GET /api/v1/snapshots | Parameter | In | Meaning | | --- | --- | --- | | node_id | query | The surface. Without paging this is its whole history, which `safegrd verify` walks to check the attestation chain. | | org_id | query | Only this organization, which must be one of the caller's | | project_id | query | Only this project's surfaces | | limit | query | Ask for a page of this many, newest first (default 50, at most 200). With limit or cursor the response is a Page; without either it is the whole list as a bare array. | | cursor | query | next_cursor from the previous page. Opaque; a key, not an offset, so rows arriving meanwhile never repeat or skip one. | | Response | Meaning | Body | | --- | --- | --- | | 200 | Snapshots, each a summary: its per-table, per-file and per-folder statistics are in GET /api/v1/snapshots/{id} | | | 400 | Refused. The body says why. | Error | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### One snapshot's recorded metadata: digests, counts, lock, Threat Shield verdict GET /api/v1/snapshots/{snapshot_id} | Parameter | In | Meaning | | --- | --- | --- | | snapshot_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The snapshot | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### A surface's Fire Drill reports, with assertions and signatures GET /api/v1/verifications | Parameter | In | Meaning | | --- | --- | --- | | node_id | query | The surface. Without paging this is its whole history, which `safegrd verify` walks to check the attestation chain. | | org_id | query | Only this organization, which must be one of the caller's | | project_id | query | Only this project's surfaces | | limit | query | Ask for a page of this many, newest first (default 50, at most 200). With limit or cursor the response is a Page; without either it is the whole list as a bare array. | | cursor | query | next_cursor from the previous page. Opaque; a key, not an offset, so rows arriving meanwhile never repeat or skip one. | | Response | Meaning | Body | | --- | --- | --- | | 200 | Fire Drill reports | | | 400 | Refused. The body says why. | Error | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### One Fire Drill's whole record, the surface it drilled, and its public certificate if one is published GET /api/v1/verifications/{verification_id} | Parameter | In | Meaning | | --- | --- | --- | | verification_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The drill | object | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | | 404 | Refused. The body says why. | Error | ### A published drill certificate: a statement, and an Ed25519 signature over the statement's bytes. Needs no token. GET /api/v1/certificates/{certificate_id} (no token needed) | Parameter | In | Meaning | | --- | --- | --- | | certificate_id (required) | path | The id in the certificate's /c/ link | | Response | Meaning | Body | | --- | --- | --- | | 200 | The signed statement | object | | 404 | Refused. The body says why. | Error | ### Counts of surfaces, snapshots, drills and open anomalies for the caller's scope GET /api/v1/summary | Parameter | In | Meaning | | --- | --- | --- | | org_id | query | Only this organization, which must be one of the caller's | | project_id | query | Only this project's surfaces | | Response | Meaning | Body | | --- | --- | --- | | 200 | Counts | Summary | | 401 | Refused. The body says why. | Error | | 403 | Refused. The body says why. | Error | ### List the buckets a project writes every backup to beside its own GET /api/v1/projects/{project_id}/copies Each bucket's name, address, lock and retention, whether SafeGrd holds its key, and the latest lock of what it holds. Never a secret. Adding or changing one needs a console session of an owner or an admin. | Parameter | In | Meaning | | --- | --- | --- | | project_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The project's buckets, under copies | object | | 403 | Refused. The body says why. | Error | ### Read one restore asked for in the console GET /api/v1/restores/{restore_id} State is requested, backing_up, running, completed, failed, cancelled or expired. runs_on is host, safegrd, or command, when the console shows the command to run on a host that holds the key. | Parameter | In | Meaning | | --- | --- | --- | | restore_id (required) | path | string | | Response | Meaning | Body | | --- | --- | --- | | 200 | The restore | object | | 404 | Refused. The body says why. | Error | ## Bodies Lists come back newest first. If you pass `limit` or `cursor`, you get a `Page`. Without either, you get the whole list as a bare array. ### AddGuide | Field | Type | Meaning | | --- | --- | --- | | can_finish | boolean | False when this caller, plan or server cannot complete it; note says why. | | commands | array | | | console_url | string | | | kind | string | | | note | string | | | steps | array | | ### Error | Field | Type | Meaning | | --- | --- | --- | | error (required) | string | | ### Page One page of a list, newest first. next_cursor is absent on the last page. | Field | Type | Meaning | | --- | --- | --- | | items (required) | array | | | next_cursor | string | | ### Summary Counts for the caller's scope, as the console's tiles show them. | Field | Type | Meaning | | --- | --- | --- | | drills | integer | | | drills_passed | integer | | | flagged_surfaces | integer | | | nodes | integer | | | open_anomalies | integer | | | snapshots | integer | | | surfaces | integer | | The same operations are available as MCP tools ([AI agents](https://safegrd.dev/docs/mcp)). For everything an agent can do from the command line, see [the command reference](https://safegrd.dev/docs/cli). --- Source: https://safegrd.dev/docs/agents # Agent guard Agent guard takes a locked snapshot before your coding agent runs a destructive command. `safegrd guard` backs up a surface, waits until the snapshot is uploaded and locked, and then runs the command. Add it as a hook in your coding agent, and it runs before every shell command that matches the destructive list. If the agent drops a table or resets a database, you have a snapshot from just before. ## Run a command behind a snapshot ``` safegrd guard --surface app-primary -- psql "$DATABASE_URL" -c 'DROP TABLE sessions' safegrd guard -- npx prisma migrate reset ``` Guard backs up the surface you name with `--surface`. Without it, guard uses the only surface in the config, or the database `SAFEGRD_DATABASE_URL` names if the config lists none. If the backup fails or the snapshot is not locked, the command does not run. ## Undo what the command did Guard prints the snapshot it took. To bring back a table the command dropped or emptied, restore that table from the snapshot. A dropped table is created again as the snapshot defined it, with its sequences, constraints and indexes. An emptied table has its rows loaded back in. The database's other tables are left as they are. ``` safegrd restore --snapshot snap-20261005-120428-524927 --table public.sessions --target env:DATABASE_URL ``` For a command that changed more than a few tables, restore the whole snapshot into an empty database instead ([Databases](https://safegrd.dev/docs/surfaces/databases)). | Exit code | Meaning | | --- | --- | | the command's own | The snapshot is locked and the command ran. | | 3 | Guard did not run the command: the backup failed or the snapshot is not locked. | | 1 | A usage or configuration error. | A snapshot counts as locked when it was written under Object Lock with a retention date in the future. Storage on the host's own disk and a bucket with `worm_mode: NONE` cannot lock, so guard refuses to run the command there unless you pass `--allow-unlocked`. Guard prints everything to stderr and leaves stdout to the command. ## Reuse a recent snapshot An agent often runs several destructive commands in a row. Pass `--max-age` and guard reuses a locked snapshot of the surface younger than that, and tells you which one. If there is none, it backs up as usual. An unlocked snapshot is never reused, so with `--allow-unlocked` every command takes a new backup. ``` safegrd guard --max-age 30m -- psql "$DATABASE_URL" -c 'TRUNCATE sessions' ``` `--check-only` takes no backup. Guard refuses the command (exit 3) unless a locked snapshot younger than `--max-age` exists. Use it when the surface's daemon or SafeGrd backs it up, and the host the agent runs on should only check. A snapshot counts if it belongs to this surface, is complete and is still under Object Lock, as `safegrd list --json` reports it. ``` safegrd guard --max-age 1h --check-only -- terraform destroy ``` ## The destructive list `safegrd guard --list` prints the rules. To test a command without taking a backup, run `safegrd guard --matches ""`. It exits 0 if the command matches and 1 if it does not. Guard matches the whole command line, so SQL inside quotes counts. | Rule | Matches | | --- | --- | | SQL DROP | `DROP TABLE`, `DATABASE`, `SCHEMA`, `VIEW`, `INDEX`, `COLUMN` and the other object types | | SQL TRUNCATE | `TRUNCATE users`, `TRUNCATE TABLE orders` | | SQL DELETE without WHERE | `DELETE FROM users;` | | dropdb | `dropdb app` | | terraform destroy | `terraform destroy`, `terraform apply -destroy`, and `tofu` | | prisma | `prisma migrate reset`, `prisma db push --force-reset` | | supabase db reset | `supabase db reset` | | rails | `db:drop`, `db:reset`, `db:schema:load`, `db:purge` | | django | `manage.py flush`, `manage.py reset_db` | A command that only mentions one of these, such as `grep "DROP TABLE" schema.sql`, also matches. That costs you one extra snapshot. ## Hooks Install the CLI and enroll the host first. Then add the hook to the project, naming the surface the agent works on. The recipes for each tool are below. With `--hook`, guard reads the hook's JSON on stdin and answers in the tool's own format. A command that does not match is let through at once, with no backup. A command that matches goes through if the surface has a locked snapshot from the last 30 minutes (`--max-age` changes the window). Otherwise guard starts a backup and waits up to 20 seconds for it (`--wait`). If the backup finishes in time, the command runs. If not, guard blocks the command and tells the agent to run it again shortly. The backup keeps going in its own process, and the retry goes through once the snapshot is locked. So a backup that finishes within 20 seconds costs the agent that wait, and a longer one costs it one retry. You use the same hook line whatever the size of the database. To let a command through, guard exits 0. For Claude Code, it also prints the snapshot that covers the command and the restore command, and the agent sees both. Codex gets no output, and Cursor gets `{"permission":"allow"}`. Letting a command through is not the same as approving it. The tool's own permission settings still decide. To block a command, guard prints the tool's deny answer with the reason. If a backup fails or is not locked, the command is blocked and the agent is told why. With `--check-only`, guard takes no backup and blocks the command unless there is a recent locked snapshot. If guard itself cannot run (the binary is missing from the agent's `PATH`, or a flag is wrong), the agent's tool would let the command through. Every recipe below ends with a fallback that blocks the command in that case. ### Claude Code Add this to `.claude/settings.json`. If you have the [Claude Code plugin](https://safegrd.dev/docs/mcp#remote) installed, ask Claude to set up the SafeGrd guard hook and it adds this for you. ``` { "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "command": "safegrd guard --hook claude-code --surface app-primary || exit 2", "timeout": 60 } ] } ] } } ``` ### Cursor Add this to `.cursor/hooks.json`: ``` { "version": 1, "hooks": { "beforeShellExecution": [ { "command": "safegrd guard --hook cursor --surface app-primary || echo '{\"permission\":\"deny\",\"agent_message\":\"safegrd guard could not run\"}'", "timeout": 60 } ] } } ``` ### Codex Add this to `.codex/hooks.json`: ``` { "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "command": "safegrd guard --hook codex --surface app-primary || exit 2", "timeout": 60 } ] } ] } } ``` ## Check the setup ``` safegrd doctor --agent-proof ``` This checks what an agent on this host could reach, and prints the fix for each check that fails. Add `--json` for a result you can share. | Check | Passes when | | --- | --- | | Bucket in another account | The bucket and the database are with different providers, or the bucket is SafeGrd's hosted storage. If both are with one provider, the config cannot tell the accounts apart, and the check says so. | | Storage key cannot delete | The bucket refuses to let this host's key delete. The check tries the delete on an object path that no snapshot uses. | | Object Lock in compliance mode | `worm_mode` is `COMPLIANCE` and the bucket has Object Lock enabled. | | Drill passed in 7 days | SafeGrd has a passed Fire Drill for each surface from the last 7 days. | | Agent uses a personal token | The agent configurations on this host give SafeGrd a personal access token (`sg_pat_`), or use the local `safegrd mcp`. | --- Source: https://safegrd.dev/docs/account # Organizations and billing Billing, quotas and members belong to an **organization**. Inside it, **projects** keep environments apart, such as production, staging or a client. Each host belongs to one project. ``` safegrd whoami safegrd org list safegrd org members safegrd project list ``` `org list` prints your organization's profile, its current plan and how much of its quota is used. To enroll a host into a particular project, run `safegrd enroll --project `. Without `--project`, the host goes into the organization's default project. If your credential can see more than one organization, name the one you mean with `--org`. ## Signing in | Command | When | | --- | --- | | safegrd login | Interactive. Opens a browser to confirm the session. `--no-browser` prints the URL instead. | | safegrd login --token sg_pat_... | CI and headless hosts. A personal access token, non-interactive. | | safegrd enroll --token ... | Registering a _host_ rather than a person. A personal access token registers a new host. A node token belongs to one host and can only re-enroll that host. | You create and revoke tokens in the console under [Tokens](https://safegrd.dev/dashboard#keys). To create a token, or to authorize `safegrd login` in the browser, you enter your password again. Someone who finds your console open cannot make a token without it. SafeGrd also emails you when a token is created in the console. If you get that email for a token you did not create, revoke the token and change your password. ### Two-factor sign-in Turn it on under [Settings](https://safegrd.dev/dashboard#settings), in _Two-factor sign-in_: - Enter your password. An account made by a Google or GitHub sign-in has no password. If you have not signed in recently, the console asks you to sign in with the provider again. - Scan the QR code with an authenticator app (1Password, Google Authenticator, Authy or similar), or type the key it shows. - Enter the 6-digit code the app shows. - Save the ten recovery codes. They are shown only once. If you lose the app, each code signs you in one time. _New recovery codes_ replaces the whole set. Turning two-factor on signs out every other session. From then on, sign-in asks for a code after the password. The console also asks for a code before you: - delete anything - create an access token - authorize `safegrd login` - cancel a subscription - change an organization's sign-in rules One code covers those actions for five minutes. After five codes in one minute, you have to wait a minute before the next. A password reset does not turn two-factor off, except on an account whose address was never verified. When you enter a code at sign-in, you can tick _Remember this browser for 30 days_. For the next 30 days, that browser asks only for the password or the provider. Signing out does not forget the browser, so leave the box unticked on a shared computer. Every browser asks for the code again after a password reset, after you sign out everywhere, and after you turn two-factor off or on again. A remembered browser still asks for a code before you delete something or take another of the actions listed above. An owner or admin who has two-factor on can require it for every member of the organization, from the same card. A member who has not set it up is sent to the setup step at sign-in and cannot use the console until they finish. While the rule is on, members cannot turn two-factor off. _Ask for a code at every sign-in_ stops members from using remembered browsers, including browsers they had already chosen to remember. Personal access tokens and node tokens are never asked for a code, so the CLI and enrolled hosts keep working when you turn the rule on. ### Google and GitHub If the server has them configured, the sign-in and sign-up pages offer Google and GitHub. SafeGrd accepts only an address the provider has verified (for GitHub, the primary address). What happens next depends on the address. - If SafeGrd already has an account for the address, you sign in to that account and the provider is linked to it. Later sign-ins follow the link even if your address at the provider changes. For a Gmail address, or a Google Workspace account on the address's own domain, the link is made straight away. For any other address, SafeGrd emails a link to confirm, and the provider is linked once you open it and press _Link_. If the account's address was never verified, the provider is not linked. Use [Forgot password](https://safegrd.dev/forgot-password) first. That link verifies the address and ends the sessions, access tokens and two-factor set up before it. Then the provider links. - If SafeGrd has never seen the address, it creates a new account with its own organization, as a sign-up would. The account has no password. - An invitation does not create an account. To join with Google or GitHub, sign in with the invited address first, then open the invitation link and accept. An account without a password confirms sensitive actions (creating a token, authorizing `safegrd login`, turning two-factor on) with a 6-digit code mailed to its address. The code works once, for 10 minutes. With two-factor on, a code from the authenticator confirms these actions instead, and turning two-factor off takes that code alone. Signing in with the provider again does not count as confirmation, because a browser already signed in to Google or GitHub comes straight back without asking. To add a password, press _Email me a link_ under Settings, or use [Forgot password](https://safegrd.dev/forgot-password) on the sign-in page. After that, you can sign in with either the password or the provider. _Sign-in methods_ in Settings lists the Google and GitHub accounts linked to yours. _Unlink_ removes one. If you later sign in with it from the same verified address, it links again, after the email confirmation where one is needed. You cannot unlink a provider in two cases: the provider created your account and you have not set a password yet, or your organization requires that provider. _Sign out everywhere_ ends every console session, including this one, and makes every remembered browser ask for the code again. Access tokens keep working, so review them under Tokens. ### Require Google or GitHub An owner or admin can require members to sign in with Google, or with GitHub, under _Two-factor sign-in_ in Settings. You have to be signed in that way to set the rule. Password sign-in then stops working for the organization's members, and a session signed in another way is sent back to sign in. Personal access tokens and node tokens are not affected. Two-factor still applies: with it on, a provider sign-in asks for a code too. An account signs in one way at a time. So a person cannot belong to one organization that requires Google and another that requires GitHub. SafeGrd refuses the invitation, or the rule, that would cause that. With Google required, you can name the Google Workspace domain the accounts have to belong to, such as `example.com`. SafeGrd then refuses a sign-in from another domain or from a consumer Google account, and an invitation to an address outside the domain. Your own address and every member's have to be in the domain before you can set it. Members of an organization that requires a provider get a session of 1 day instead of 7. If the provider suspends an account, that account loses the console within a day, and its next sign-in goes through the provider again. ### Expiry and scope A new personal access token expires after 90 days. The console shows the date and warns you two weeks before. You can set a shorter lifetime when you create a token, but you cannot make one that never expires. Tokens created before this rule have no expiry date and keep working. You can also limit a token to one organization or one project. The token then reaches only that organization or project, whatever your account itself can reach. A token limited to a host's project can reach only that project if it leaks. ## Plans and quotas The [pricing page](https://safegrd.dev/pricing) lists each plan, what it includes and its price. The same list is at `/api/v1/plans`, and SafeGrd enforces exactly what it lists. Paid plans differ by how often and how thoroughly they test a restore. The only thing a plan meters is [hosted storage](https://safegrd.dev/docs/storage#hosted), the bytes SafeGrd keeps locked for you. Paid plans do not count surfaces (databases, file trees or mailboxes) or team members, and every paid plan backs up hourly. The free plan is for one person and one surface, backed up daily to hosted storage. Backups to your own bucket are never metered. ## The trial, and the free floor During the 14-day trial, you get Growth’s Fire Drills. If the trial ends without an active subscription, your organization moves to the free plan. On the free plan, your backups stay encrypted and locked, and a Fire Drill runs on them once a week, in memory. The [pricing page](https://safegrd.dev/pricing) lists how many surfaces it covers. Any organization without a paid plan is on the free plan, whether its trial expired or its subscription was paused or canceled. ## What lapsing does not do ### A billing failure never stops a backup. If your subscription lapses, every surface you already have keeps backing up, and everyone already in the organization keeps their access. Only _new_ surfaces and new team members are blocked. Snapshots in your own bucket stay there, under your Object Lock, whatever happens to the subscription or to SafeGrd. With a customer-managed key, you can read them with your key without SafeGrd. Your plan changes when SafeGrd receives the payment provider's signed webhook. Returning from the checkout page does not change it. ## Team members Owners and admins invite people from Settings, Team members. You can invite someone as an admin, a member, a viewer or an auditor. Team members are included on every paid plan and during the trial. An auditor is someone outside the team who checks that the backups were tested. An auditor can sign in and see the organization's drill records, each drill's signed certificate and the Evidence Pack. They cannot see surfaces, hosts, snapshots, storage, members or settings, and they get no button that starts or changes anything. An auditor's personal access token reads the same records, so they can run `safegrd verify history` to verify the chain on their own machine. Only an owner or admin can invite an auditor. How an invitation works: - The invited address gets an email with a link that works for 7 days. Until the person accepts, the list shows the address as _Invited_, and it has no access to the organization. - Someone who already has a SafeGrd account accepts while signed in as that address. Someone without one creates an account on the link's page, and joins with no organization of their own. - _Resend_ mails a new link, and the old one stops working. _Revoke_ cancels the invitation. The person can also decline from the link. - A plan without team members refuses new invitations. It also refuses an invitation sent before the plan changed, when the person tries to accept it. Members who already joined stay. Two switches on Settings, Security control what members can do. Owners and admins can always do both. - _Members can invite people_ lets a member invite members and viewers, but never an admin. It is off in a new organization. - _Members can create projects_ is on in a new organization. An organization created before 17 October 2026 has both switches off until an owner or admin turns them on. ## The owner The account that created an organization owns it. You cannot hand ownership to another member yet. Each account owns one organization, so after you delete it, the same account cannot create another. ## Invoices and the card on file Under _Billing_, the owner sees the card that renewals are charged to and every invoice, with a PDF to download. To replace the card, click _Update_, which opens the payment provider's form. The payment provider emails a receipt for each charge. It issues no PDF for a $0 invoice. ## Deleting an organization - If your backups are on hosted storage, copy the ones you want to keep with `safegrd export`. After the organization is deleted, you cannot download them. - Decommission every host under _Nodes_, and cancel the subscription under _Billing_. SafeGrd refuses the delete until both are done, and tells you which one is left. - Under _Settings_, choose _Delete organization_ and type its name. The organization is removed from SafeGrd straight away, and you are signed out. Backups already written stay in their storage. We keep the organization's records for 12 months so it can be restored, then erase them. To have them erased sooner, write to support@safegrd.dev, and we erase them within 30 days ([Privacy Notice](https://safegrd.dev/privacy)). --- Source: https://safegrd.dev/docs/troubleshooting # Troubleshooting Start with these three commands. They answer most questions before you need to read a log. | Command | What it tells you | | --- | --- | | safegrd doctor | Checks config permissions, whether the key pair is intact, whether the sink answers and Object Lock works on it, whether SafeGrd's server answers, the host's clock skew, and whether each surface opens with the credential its backup will use. On an enrolled host the result also shows in the console. Add `--json` for CI. | | safegrd config validate | Checks config syntax, permissions and surface definitions without running anything. | | safegrd daemon status | For each surface: what is due, what is failing, and when it last succeeded. | ## Common failures ### "Bucket does not support Object Lock" Create a new bucket with Object Lock and versioning turned on. Most providers let you turn on Object Lock only when you _create_ a bucket. If your provider has no Object Lock at all (DigitalOcean Spaces, for example), set `worm_mode: "NONE"`. Backups to that sink are then not locked, and anyone with a delete key can remove them. ### A refused `worm_mode` Write the value in uppercase: `COMPLIANCE`, `GOVERNANCE` or `NONE`. ### "credential_held is no longer read" Your config was written for an older CLI. A surface now says where its credential comes from in one `credential` block, and the error names the replacement for the old key: - `credential_held: true` becomes `credential: {from: safegrd}`. - `database_url_env` and `password_env` become `credential: {from: env, name: VARIABLE}`. - `credential_command` becomes `credential: {from: command, run: "..."}`. See [Configuration](https://safegrd.dev/docs/config). ### Config permissions rejected Run `chmod 600 ~/.safegrd/config.yaml`. The file holds connection strings, so on POSIX hosts the CLI refuses to load it if its group or other users have any access to it. ### Restore cannot find a key On a recovery host, copy the key file over and point `--key-path` at it. The CLI looks for the private key in `key_path`, `SAFEGRD_PRIVATE_KEY`, `--key-path` or `--private-key`. The key is never stored in `config.yaml`. ### My snapshots vanished Under Object Lock they are almost certainly still there. A `DELETE` writes a delete marker that hides them. Run `safegrd undelete --all` to remove the markers. See [Storage and retention](https://safegrd.dev/docs/storage). ### IMAP certificate errors on a self-hosted server Point the surface's `ca_file` at your internal CA bundle. You can also pass `--email-ca-file` to a one-off backup, or set `SAFEGRD_EMAIL_CA_FILE`. Certificate checks cannot be turned off. ### Clock skew warnings Run NTP on every host that takes backups. The attestation record is ordered by time, and signatures are checked against that order. `doctor` reports the drift it sees. ## Client tools A database backup runs the database's own dump tool on your host: `pg_dump`, `mysqldump` or `mariadb-dump`, or `mongodump`. Run `safegrd doctor` after you install or upgrade a client. It names the tool each surface will use. ### "the newest pg_dump here is 15.8 (/usr/bin/pg_dump), older than the PostgreSQL 17 server" Install a PostgreSQL client of the server's major version or newer. `pg_dump` refuses to dump a server newer than itself, and a newer `pg_dump` dumps older servers correctly. On Debian and Ubuntu, add the PostgreSQL apt repository (`apt.postgresql.org`) and install `postgresql-client-17`. On macOS, run `brew install libpq`. SafeGrd looks in your `PATH`, `/usr/lib/postgresql/*/bin`, `/usr/pgsql-*/bin` and Homebrew's `libpq` and `postgresql@*`, and picks the newest `pg_dump` it finds. To use one installed somewhere else, set `SAFEGRD_PG_DUMP` to its path. ### "no pg_dump found" or "no pg_dumpall beside ..." Install the PostgreSQL client package. It includes both tools. SafeGrd reads the database's roles with `pg_dumpall`, and it uses the one in the same directory as the `pg_dump` it picked. ### "no mysqldump or mariadb-dump found" Install `mysql-client` for a MySQL server, or `mariadb-client` for MariaDB. You can also set `SAFEGRD_MYSQLDUMP` to the tool's path, and `SAFEGRD_MYSQL` to the client a restore uses. ### "dumping a MariaDB server with mysqldump ..." This is a warning, and the backup still runs. Install the client from the same project as the server. When both clients are installed, SafeGrd picks the matching one. MySQL's `mysqldump` and MariaDB's `mariadb-dump` each write their own project's SQL best. A dump made with the other project's tool may need edits before the server restores it. ### "no mongodump found" Install the MongoDB Database Tools (`mongodb-database-tools`). They come in their own package, separate from the server and the shell. You can also set `SAFEGRD_MONGODUMP` to the tool's path. A restore needs `mongorestore` from the same package, or `SAFEGRD_MONGORESTORE` set to its path. ## SafeGrd's server is unreachable Backups keep running. The CLI prints a warning to stderr and writes the backup to your sink as usual. When the server answers again, the host sends its report. ## Getting help Include the output of `safegrd version` in any report. It prints the version, commit and build date. Attach the output of `safegrd doctor --json` too: it is the most useful thing you can send. File issues against the CLI at [github.com/safegrd/cli](https://github.com/safegrd/cli). For a problem in the console, open _Help_, click _Copy diagnostics_ and paste the result into your email to [support@safegrd.dev](mailto:support@safegrd.dev). It includes the console's address, the organization and plan, your role, the surfaces by kind, the browser, the time and the last five error notices from your session. It includes no token, key or password.