Daemon
The daemon runs the backups in your config file on schedule. On each pass it reads the config, finds the surfaces that are due, locks each one and backs it up. It also reports the host's health to SafeGrd, so you are alerted if the host stops responding.
Run it
The first pass backs up every surface that is due. A new surface is due at once:
$ safegrd daemon run --once SafeGrd daemon dev started Surfaces: 1 State directory: /home/safegrd/.safegrd Surface shop (postgres) is due for backup. [shop] Epoch e202610-06bca287 opened (first backup). Every table is uploaded once this month. [shop] 3 tables, 70,240 rows, 3.1 MiB read, 386.9 KiB stored in 2 packs, 532ms. [shop] Snapshot snap-20261018-024047-450e5d complete. Locked until 2026-11-02: the month's first backup is kept as its monthly copy. Later backups follow the surface's retention. [shop] That date is capped: a backup is locked no longer than the objects it is built from, and no longer than a trial runs. Backed up surface shop
$ safegrd daemon status SafeGrd daemon status Daemon ID: afc6de1d State dir: /home/safegrd/.safegrd SURFACE ID TYPE SCHEDULE LAST SUCCESS NEXT DUE STATUS shop postgres @daily 2026-10-18 02:40 UTC 2026-10-19 02:40 UTC OK
daemon status --json prints the same, with each surface's last run and next due time as timestamps.Install it as a service
The service runs the same binary with the config you installed it from (the one
--config names, or ~/.safegrd/config.yaml). It backs up exactly
what safegrd daemon run backs up in your shell.
The daemon keeps its state and locks in ~/.safegrd of the user who ran
install. For a system service installed with sudo, that is
/root/.safegrd. A system service sees the filesystem as read-only, except for
that directory and a local sink. install creates both if they do not exist yet.
A user service stops when you log out unless you run
loginctl enable-linger $USER once. To restart the service, run
safegrd daemon restart (add --user for a user service).
safegrd daemon uninstall removes the unit file and prints the command that stops
a service that is still running.
Install it as a system service or as a user service, not both. Two daemons writing to the same sink as different users leave directories that one of them cannot write to.
Config changes and upgrades
The daemon reads the config again at the start of each pass if the file changed. A surface
you add by hand or with safegrd claim (details)
is backed up without a restart. If the changed config does not load, the daemon logs it and
keeps the config it had.
Changes to metrics_addr, interval, updates and SQLite
followers take effect at the next safegrd daemon restart. So does an upgrade you
install by hand. A running daemon keeps running the binary it started with and logs once that
the file was replaced, so run safegrd daemon restart after you upgrade. An update
from the console needs no restart (below).
Schedules
Each surface's schedule sets how often it backs up. The daemon's
interval is only how often it wakes up to look for work. The daemon adds
about 10% jitter, so a fleet of hosts does not hit the same bucket on the same minute.
| Form | Meaning |
|---|---|
| @hourly, @daily, @weekly | The usual shorthands. |
| 6h, 90m | A plain duration between runs. |
| 2d | A whole number of days. |
| at 02:00 | Once a day, at that time. |
| 0 3 * * 1-5 | A cron line: minute, hour, day of month, month, day of week. This one is 03:00 on weekdays. |
A schedule cannot run more often than once an hour. safegrd config validate
refuses a shorter one, and a cron line whose two closest runs are less than an hour apart.
Under compliance-mode Object Lock, each snapshot stays locked until its lock date, so frequent
backups add up: a 30-minute schedule with 30-day retention would keep 1,440 snapshots.
Times of day and cron lines are in UTC. Set clock: local on a surface to read
them, and its blackout windows, in the host's time zone. The surface backs up at the first
such time after its last backup. For a cron line, Overdue uses the longest gap between two
runs: three days for 0 3 * * 1-5, Friday to Monday.
Owners and admins can also set a surface's schedule and retention in the console: open the
surface's Details under Surfaces and click Edit. The daemon picks up the new settings at its
next check-in and uses them instead of the config's until someone clears them in the console.
The config file on the host does not change. The daemon logs this when the settings change and
once each time it starts, and safegrd daemon status marks such a schedule
(console). Until the daemon reports that it runs a setting, the console shows it
as waiting.
Use blackout to list times of day when the daemon starts no scheduled backup for
a surface, for example blackout: ["22:00-02:00"]. The times are in UTC, or on the
surface's clock. A backup that falls due inside a window waits until the window
ends, and the daemon logs that it is waiting and until when. A backup that is already running
when a window opens runs to the end. Back up now from the console is not held back.
Overdue waits longer by the total length of the surface's windows.
safegrd config validate checks each window.
To pause a surface's schedule, click Edit on its row in the console, then Pause the schedule. Give a reason and, if you want, a time to resume. While the surface is paused, no scheduled backup or drill runs and it is not reported Overdue. Back up now and Drill now still run. WAL archiving goes on, and its point-in-time drill waits with the others. The daemon logs the pause at its next check-in and keeps it while SafeGrd's server cannot be reached.
Click Resume on the row to end the pause. The backup that is due then runs at the next check-in, and Overdue counts from the resume. Pausing works the same way for a surface SafeGrd backs up itself. The activity log records who paused it and why. The 30-day history shows the paused days as paused, so they do not count as missed, and the worst gap leaves the paused time out (recovery figures).
Set skip_if_unchanged: true on an incremental surface to skip writing a backup
when nothing changed since the last one: no new data, and no file or table added, changed or
removed. The run still reads the whole surface to find that out. The daemon logs the skip and
the schedule starts again from that run. The console shows the surface as unchanged since its
last backup instead of Overdue. Drills keep restoring the last backup.
This works on your own bucket and on local storage. On hosted storage every run writes a backup, and the daemon says so when it starts.
On every pass, the daemon works out each surface's next run from the host's clock. If the
clock goes back (a bad NTP step, or a VM restored from an image), a surface whose last backup
is now in the future backs up at once instead of waiting for the clock to catch up. If the
clock moves by more than interval between two passes, the daemon logs how far
and in which direction. A machine that was suspended shows up as a step forward.
The daemon block
| Key | What it does |
|---|---|
| interval | How often the daemon looks for work. Default 5m. |
| state_dir | Where the daemon keeps state, locks and logs. Do not put it on a network filesystem. The lock on a backup in progress needs flock to work. |
| metrics_addr | The address to serve Prometheus metrics on, at /metrics. See Metrics and pings. |
| ping_url | A check URL the daemon pings after every pass. See Metrics and pings. |
The daemon backs up one surface at a time. It tries a failed backup 3 times in all, 5 minutes apart, then waits for the surface's next scheduled run. It logs plain text to stdout and stderr, which systemd and launchd keep.
Credentials the daemon needs
A daemon cannot ask you for a password. Instead, each surface's credential block
says where to read its secret:
from: envreads the environment variable named inname.from: commandrunsrun(your own secret manager) and reads the value from its output.from: filereads the file atpath.from: safegrd: SafeGrd keeps the secret encrypted and gives it only to this host when the surface backs up.
The first three never send the secret to SafeGrd.
- For a database, the secret is the whole connection URL. For a mailbox, it is the password.
A mailbox that names no credential uses
SAFEGRD_EMAIL_PASSWORD. - If the block names something the daemon cannot read (an unset variable, a failing command or a missing file), that surface fails. The daemon does not fall back to anything else.
- A
from: commandruns under/bin/shas the daemon's user, with a 30-second limit. If it fails, that surface fails with the command's error output, and the other surfaces carry on. It does not fall back to an environment variable. - A service does not inherit your login shell. Run
safegrd doctor, which resolves every surface's credential the way the daemon will. Then run it again as the service's user. A secret manager you are signed in to may not be signed in for that user. - The config file must be readable and writable only by its owner (0600), so nobody else can change these commands or read the credentials.
An environment variable for the service
The service that daemon install writes does not see variables you export in a
shell. Add them with a systemd drop-in, which stays in place when the unit is reinstalled:
sudo systemctl edit safegrd
# in the editor:
[Service]
Environment=APP_DATABASE_URL=postgres://backup:PASSWORD@db.internal:5432/app
sudo systemctl restart safegrd
Any user on the host can read an Environment= line with systemctl show.
On a shared host, use from: file with a file only the daemon's user can read
(0600), or from: safegrd.
Before and after a backup
You can run your own command before and after each backup of a surface, for example to flush a cache, or to pause a writer and resume it afterwards:
- If
pre_backupfails, the daemon skips that backup and counts a failure for the surface. The hook's output goes to the daemon's log. Setpre_backup_on_errorto change this.retryruns the hook again after 10 seconds and then after 20 before skipping the backup.continuetakes the backup anyway and logs a warning. The default isskip. post_backupruns after every attempt, so whateverpre_backuppaused is resumed even if the backup failed. It getsSAFEGRD_BACKUP_STATUS(successorfailed) andSAFEGRD_SNAPSHOT_ID.post_drillruns after each Fire Drill the daemon runs, for example to run a smoke test or warm a cache. It getsSAFEGRD_DRILL_STATUS(passed,shallow,failedorblocked) andSAFEGRD_SNAPSHOT_ID. If the sandbox still holds the restored data, it also getsSAFEGRD_SANDBOX_URL. That is a throwaway PostgreSQL on the host, or a sandbox kept after a failed drill. If the hook fails, the daemon logs it and nothing else changes.- All three hooks run under
/bin/shas the daemon's user, with a 10-minute limit each. They live in the config file, which is another reason it must be readable only by its owner.
What the console sees
Each surface in the config shows up in the console on its own, under the host you enrolled, from the first time the daemon sees it. The host's token covers all of its surfaces, so there is one credential per host, and rotating it covers them all. A new surface counts against your plan like any other.
On every tick, the daemon checks in for each surface. That lets the console tell the daemon has stopped apart from this one surface is failing:
- Silent: the daemon has missed three check-ins (and at least 15 minutes). Nothing is backing up that host.
- Overdue: a surface has gone twice its schedule, plus its
blackout windows, without a backup. This also applies to hosts
that run
daemon run --oncefrom cron. Those do not count as daemons unless each pass names its schedule with--interval, for exampledaemon run --once --interval 5mfrom a cron line that runs every 5 minutes. - Failing: the last attempt failed, with the error the daemon saw.
- Unreachable: on three ticks in a row, the daemon could not open the surface’s database or reach its sink. The row shows since when, and the error.
Checks between backups
If a database goes away at 09:00, the console shows it a few ticks later, without waiting
for the next scheduled backup. On each tick, before it checks in, the daemon opens the
database of each PostgreSQL, MySQL and MongoDB surface and runs SELECT 1 (or the
engine’s ping). It also checks that the sink answers, with a HeadBucket on
your own bucket or a look at the local directory. Each check gives up after 10 seconds.
The checks read nothing else and write nothing, with one exception. For a surface that names
erasures:, the daemon moves the waiting rows of safegrd.erasures into
the surface’s storage (the erasure record).
If that move fails, it counts as a sink that does not answer.
These surfaces are checked only when they back up:
- A surface whose credential SafeGrd holds, because the daemon fetches that credential for a backup or a drill and for nothing else.
- A surface whose credential comes from
credential.from: command, which would otherwise run that command every tick.
The daemon does not check hosted storage, or a bucket whose access key SafeGrd holds. A surface with nothing to check is shown as its last backup left it.
Back up now, under a surface’s Details in the console, tells the daemon to back the surface up at its next check-in. If the previous backup was less than an hour ago, the daemon skips the request, so locked storage does not fill with redundant snapshots that cannot be deleted.
Alerts
On any plan, add a Slack, Discord or plain webhook URL under Settings → Alerts in the console. Paid plans also email the organization's owners and admins.
You get an alert for a failed backup, a failed Fire Drill, a drill that cannot run because the key is missing, a silent host, and an overdue or unreachable surface. You get another when the backup, host or surface recovers. Each is sent once, not on every check. Alerts lists every event and what each message contains.
Metrics and pings
Set daemon.metrics_addr: "127.0.0.1:9847" and the resident daemon serves
Prometheus metrics at http://127.0.0.1:9847/metrics. Each scrape reads the
state the daemon wrote after its last backup. Every series except the first two has
surface and type labels:
| Metric | What it is |
|---|---|
safegrd_daemon_info | 1, with the daemon's version |
safegrd_daemon_state_updated_timestamp_seconds | When the daemon last wrote its state |
safegrd_surface_last_attempt_timestamp_seconds | When the last backup started |
safegrd_surface_last_success_timestamp_seconds | When the last backup succeeded |
safegrd_surface_consecutive_failures | Failed backups since the last success |
safegrd_surface_next_due_timestamp_seconds | When the next scheduled backup is due |
safegrd_surface_last_backup_duration_seconds | How long the last successful backup took |
safegrd_surface_last_backup_size_bytes | What it stored, encrypted |
safegrd_surface_last_drill_timestamp_seconds | When the last Fire Drill ran |
safegrd_surface_drill_passed | 1 when the last Fire Drill passed, else 0 |
A timestamp for something that has never happened is left out instead of written as 0. To alert when the last successful backup is more than a day old:
time() - safegrd_surface_last_success_timestamp_seconds > 86400
Bind it to 127.0.0.1 unless the scraper runs on another machine, because the
metrics include your surface names.
To use a check service such as Healthchecks.io or Cronitor, give a surface a
ping_url. The daemon posts to URL/start before each backup and to
URL after it. If the backup failed, or SafeGrd did not record it, the daemon
posts to URL/fail instead. Each post carries one line about what happened as its
body.
The daemon pings daemon.ping_url after every pass, so the check alerts you when
the daemon stops running. Both settings take env:VAR or file:/path,
because anyone who has a check URL can report to that check. If a ping fails, the daemon
prints a warning to stderr and the backup's result stands.
daemon:
metrics_addr: "127.0.0.1:9847"
ping_url: "env:HEALTHCHECK_DAEMON"
surfaces:
- id: "orders-db"
type: "postgres"
ping_url: "https://hc-ping.com/your-check-uuid"
Fire Drills on an unattended host
When a surface is due for a drill on your plan's schedule, SafeGrd's server tells the daemon. The daemon restores the latest snapshot, checks it against the digests recorded when it was taken, and reports the result to your attestation record.
A drill needs the private key, so the daemon runs one only on a host that already has it
(through key_path, SAFEGRD_PRIVATE_KEY, or a SafeGrd-managed key).
Every surface in an organization is sealed to that one key, and the daemon never copies it
to a host. A host without the key reports Drill blocked: no key, and you are
alerted. You can then run safegrd verify wherever you keep the key.
For a PostgreSQL surface, you can give the drill a scratch database. The drill then restores the rows into it and counts them there. Without one, the snapshot is replayed in memory, unless the host can run a throwaway cluster (below).
Create the sandbox once (createdb app_drill) and give the daemon a role that
owns it. Use it for nothing else, because a drill restores into it. The daemon refuses a
sandbox that holds any table, or one that is the database the surface backs up, and it
empties the sandbox after every drill.
If your plan includes sandbox drills, a PostgreSQL surface with no
drill.sandbox_url gets a scratch database on the host. When the host has the
PostgreSQL server installed (initdb and postgres, of the source
database's major version or newer), the daemon creates a throwaway cluster under its state
directory and restores the snapshot into it. It deletes the cluster when the drill ends. You
do not need Docker.
- The cluster listens on a Unix socket only, in a directory only the daemon's user can open.
- It uses under 200 MB of memory, whatever the size of the database. Restored rows go to disk.
- It needs about as much disk as the database. The daemon checks first, and does not start a cluster that would leave less than 5% of the filesystem free (1 GiB on a disk over 20 GiB).
- PostgreSQL does not run as root, so a daemon running as root runs the cluster as the user who owns its state directory.
Look at a failed drill's sandbox
Set drill.keep_failed_sandbox: true to keep the sandbox as it was for 24 hours
when a drill restores the snapshot and then fails a check. The daemon prints where the
sandbox is, without the password:
A sandbox_url database is left alone for those 24 hours. The next drill is held
back, and the first drill after the 24 hours empties it. A throwaway cluster keeps running on
its socket until the daemon removes it. Everything stays on the host. If the restore itself
fails, it is rolled back and there is nothing to keep. The error is in the daemon's log and
in the console.
Install a PostgreSQL server for sandbox drills
On Debian and Ubuntu, install the server with apt install postgresql-16. The
postgresql-client package has no server. The server package also starts a server
of its own, which the daemon does not use. Stop it with
systemctl disable --now postgresql. On macOS, run
brew install postgresql@16.
If the daemon cannot start a cluster (no server, a server older than the database, too little disk, or an extension the snapshot uses that the server lacks), the drill runs in memory. The daemon's log says what to install.
When a drill is not the kind your plan includes, the console says why under the surface: it ran in memory because there was no sandbox, or it did not run because the host lacks the disk. A drill that does not run is not a failed backup. The daemon tries again at the next drill, and you are alerted once.
Disk the daemon uses on the host
Database and file backups stream to storage with no copy on the host. Four things do write to the host's disk. Before each one, the daemon checks that it would leave 5% of the filesystem free for everything else on it (at most 1 GiB):
- A drill of an incremental file surface restores the whole tree under the state directory, then deletes it. If there is no room, the drill does not run.
- A local sandbox drill needs about as much as the database, as above.
- A SQLite backup copies the database to
TMPDIRfirst. If there is no room, the backup fails and says so. PointTMPDIRat a larger disk. - A local sink (
storage.type: local) holds every backup. A backup stops before the free space would drop below that limit, and the error says what to free.
Updates from the console
When a new release is out, the daemon downloads and checks it, and the host's row under Hosts shows ready to install. An owner or admin clicks Update, and the daemon installs the release at its next check-in (every 5 minutes by default), between backups. Before it switches, it checks that:
- the archive matches the SHA-256 that the release's
checksums.txtlists - the release is newer than the one you run
- the new binary loads your config and the daemon's state with the flags your service passes
SafeGrd's server only tells the host which release to install. The host downloads it from
the release page (or from
daemon.updates_url), so SafeGrd's server never sends you a binary.
The host keeps the release it ran before. If the new one does not check in within two passes, or starts three times without checking in, the host switches back to the earlier one on its own, and the console says why. To go back yourself, click Go back. The earlier release is still on the host, so nothing is downloaded.
Console updates work on hosts set up with install.sh. It keeps each release in
/usr/local/lib/safegrd/versions (~/.local/lib/safegrd without
sudo) and points /usr/local/bin/safegrd at the one in use. On a host installed
before this existed, run the installer once. If you installed with Homebrew, the container
image or go install, update the usual way, and the console tells you which
command to run. To turn console updates off on a host, set daemon.updates: off.
When SafeGrd's server cannot be reached
The daemon keeps running scheduled backups and writing them to your storage. Each snapshot is
in your bucket and restores, and safegrd list shows it.
The daemon prints that it could not send the backup's record, and keeps the record. On the
first tick the server answers, the daemon sends it and the console lists the backup.
safegrd daemon status shows how many records are waiting.
If the server refuses a report (because the token expired, for example), the daemon does not
send it again. It is printed to stderr and shown as not recorded. A backup you
run by hand with safegrd backup keeps no state, so its record is not sent again.
With --surface, the backup goes through the daemon's state, and its record is.
safegrd daemon status shows how many are waiting.