Docs / Daemon

Daemon

The daemon runs the backups in your config file on schedule. On each pass it reads the config, finds the surfaces that are due, locks each one and backs it up. It also reports the host's health to SafeGrd, so you are alerted if the host stops responding.

Run it

# resident daemon
safegrd daemon run
# one reconciliation pass and exit, for cron or a Kubernetes Job
safegrd daemon run --once
# what every surface is doing, and when each last succeeded
safegrd daemon status

The first pass backs up every surface that is due. A new surface is due at once:

prod-app-01
$ safegrd daemon run --once
SafeGrd daemon dev started
   Surfaces:            1
   State directory:     /home/safegrd/.safegrd
Surface shop (postgres) is due for backup.
[shop] Epoch e202610-06bca287 opened (first backup). Every table is uploaded once this month.
[shop] 3 tables, 70,240 rows, 3.1 MiB read, 386.9 KiB stored in 2 packs, 532ms.
[shop] Snapshot snap-20261018-024047-450e5d complete. Locked until 2026-11-02: the month's first backup is kept as its monthly copy. Later backups follow the surface's retention.
[shop] That date is capped: a backup is locked no longer than the objects it is built from, and no longer than a trial runs.
Backed up surface shop
prod-app-01
$ safegrd daemon status
SafeGrd daemon status
   Daemon ID:   afc6de1d
   State dir:   /home/safegrd/.safegrd

SURFACE ID   TYPE       SCHEDULE   LAST SUCCESS           NEXT DUE               STATUS
shop         postgres   @daily     2026-10-18 02:40 UTC   2026-10-19 02:40 UTC   OK
daemon status --json prints the same, with each surface's last run and next due time as timestamps.

Install it as a service

# systemd on Linux, launchd on macOS
safegrd daemon install
# as a user agent rather than a system service
safegrd daemon install --user
# print the unit and install nothing, to review or commit it
safegrd daemon install --print

The service runs the same binary with the config you installed it from (the one --config names, or ~/.safegrd/config.yaml). It backs up exactly what safegrd daemon run backs up in your shell.

The daemon keeps its state and locks in ~/.safegrd of the user who ran install. For a system service installed with sudo, that is /root/.safegrd. A system service sees the filesystem as read-only, except for that directory and a local sink. install creates both if they do not exist yet.

A user service stops when you log out unless you run loginctl enable-linger $USER once. To restart the service, run safegrd daemon restart (add --user for a user service). safegrd daemon uninstall removes the unit file and prints the command that stops a service that is still running.

Install it as a system service or as a user service, not both. Two daemons writing to the same sink as different users leave directories that one of them cannot write to.

Config changes and upgrades

The daemon reads the config again at the start of each pass if the file changed. A surface you add by hand or with safegrd claim (details) is backed up without a restart. If the changed config does not load, the daemon logs it and keeps the config it had.

Changes to metrics_addr, interval, updates and SQLite followers take effect at the next safegrd daemon restart. So does an upgrade you install by hand. A running daemon keeps running the binary it started with and logs once that the file was replaced, so run safegrd daemon restart after you upgrade. An update from the console needs no restart (below).

Schedules

Each surface's schedule sets how often it backs up. The daemon's interval is only how often it wakes up to look for work. The daemon adds about 10% jitter, so a fleet of hosts does not hit the same bucket on the same minute.

FormMeaning
@hourly, @daily, @weeklyThe usual shorthands.
6h, 90mA plain duration between runs.
2dA whole number of days.
at 02:00Once a day, at that time.
0 3 * * 1-5A cron line: minute, hour, day of month, month, day of week. This one is 03:00 on weekdays.

A schedule cannot run more often than once an hour. safegrd config validate refuses a shorter one, and a cron line whose two closest runs are less than an hour apart. Under compliance-mode Object Lock, each snapshot stays locked until its lock date, so frequent backups add up: a 30-minute schedule with 30-day retention would keep 1,440 snapshots.

Times of day and cron lines are in UTC. Set clock: local on a surface to read them, and its blackout windows, in the host's time zone. The surface backs up at the first such time after its last backup. For a cron line, Overdue uses the longest gap between two runs: three days for 0 3 * * 1-5, Friday to Monday.

Owners and admins can also set a surface's schedule and retention in the console: open the surface's Details under Surfaces and click Edit. The daemon picks up the new settings at its next check-in and uses them instead of the config's until someone clears them in the console. The config file on the host does not change. The daemon logs this when the settings change and once each time it starts, and safegrd daemon status marks such a schedule (console). Until the daemon reports that it runs a setting, the console shows it as waiting.

Use blackout to list times of day when the daemon starts no scheduled backup for a surface, for example blackout: ["22:00-02:00"]. The times are in UTC, or on the surface's clock. A backup that falls due inside a window waits until the window ends, and the daemon logs that it is waiting and until when. A backup that is already running when a window opens runs to the end. Back up now from the console is not held back. Overdue waits longer by the total length of the surface's windows. safegrd config validate checks each window.

To pause a surface's schedule, click Edit on its row in the console, then Pause the schedule. Give a reason and, if you want, a time to resume. While the surface is paused, no scheduled backup or drill runs and it is not reported Overdue. Back up now and Drill now still run. WAL archiving goes on, and its point-in-time drill waits with the others. The daemon logs the pause at its next check-in and keeps it while SafeGrd's server cannot be reached.

Click Resume on the row to end the pause. The backup that is due then runs at the next check-in, and Overdue counts from the resume. Pausing works the same way for a surface SafeGrd backs up itself. The activity log records who paused it and why. The 30-day history shows the paused days as paused, so they do not count as missed, and the worst gap leaves the paused time out (recovery figures).

Set skip_if_unchanged: true on an incremental surface to skip writing a backup when nothing changed since the last one: no new data, and no file or table added, changed or removed. The run still reads the whole surface to find that out. The daemon logs the skip and the schedule starts again from that run. The console shows the surface as unchanged since its last backup instead of Overdue. Drills keep restoring the last backup.

This works on your own bucket and on local storage. On hosted storage every run writes a backup, and the daemon says so when it starts.

On every pass, the daemon works out each surface's next run from the host's clock. If the clock goes back (a bad NTP step, or a VM restored from an image), a surface whose last backup is now in the future backs up at once instead of waiting for the clock to catch up. If the clock moves by more than interval between two passes, the daemon logs how far and in which direction. A machine that was suspended shows up as a step forward.

The daemon block

KeyWhat it does
intervalHow often the daemon looks for work. Default 5m.
state_dirWhere the daemon keeps state, locks and logs. Do not put it on a network filesystem. The lock on a backup in progress needs flock to work.
metrics_addrThe address to serve Prometheus metrics on, at /metrics. See Metrics and pings.
ping_urlA check URL the daemon pings after every pass. See Metrics and pings.

The daemon backs up one surface at a time. It tries a failed backup 3 times in all, 5 minutes apart, then waits for the surface's next scheduled run. It logs plain text to stdout and stderr, which systemd and launchd keep.

Credentials the daemon needs

A daemon cannot ask you for a password. Instead, each surface's credential block says where to read its secret:

The first three never send the secret to SafeGrd.

An environment variable for the service

The service that daemon install writes does not see variables you export in a shell. Add them with a systemd drop-in, which stays in place when the unit is reinstalled:

sudo systemctl edit safegrd
# in the editor:
[Service]
Environment=APP_DATABASE_URL=postgres://backup:PASSWORD@db.internal:5432/app

sudo systemctl restart safegrd

Any user on the host can read an Environment= line with systemctl show. On a shared host, use from: file with a file only the daemon's user can read (0600), or from: safegrd.

Before and after a backup

You can run your own command before and after each backup of a surface, for example to flush a cache, or to pause a writer and resume it afterwards:

pre_backup: "redis-cli BGSAVE && sleep 5"
post_backup: "systemctl reload app"

What the console sees

Each surface in the config shows up in the console on its own, under the host you enrolled, from the first time the daemon sees it. The host's token covers all of its surfaces, so there is one credential per host, and rotating it covers them all. A new surface counts against your plan like any other.

On every tick, the daemon checks in for each surface. That lets the console tell the daemon has stopped apart from this one surface is failing:

Checks between backups

If a database goes away at 09:00, the console shows it a few ticks later, without waiting for the next scheduled backup. On each tick, before it checks in, the daemon opens the database of each PostgreSQL, MySQL and MongoDB surface and runs SELECT 1 (or the engine’s ping). It also checks that the sink answers, with a HeadBucket on your own bucket or a look at the local directory. Each check gives up after 10 seconds.

The checks read nothing else and write nothing, with one exception. For a surface that names erasures:, the daemon moves the waiting rows of safegrd.erasures into the surface’s storage (the erasure record). If that move fails, it counts as a sink that does not answer.

These surfaces are checked only when they back up:

The daemon does not check hosted storage, or a bucket whose access key SafeGrd holds. A surface with nothing to check is shown as its last backup left it.

Back up now, under a surface’s Details in the console, tells the daemon to back the surface up at its next check-in. If the previous backup was less than an hour ago, the daemon skips the request, so locked storage does not fill with redundant snapshots that cannot be deleted.

Alerts

On any plan, add a Slack, Discord or plain webhook URL under Settings → Alerts in the console. Paid plans also email the organization's owners and admins.

You get an alert for a failed backup, a failed Fire Drill, a drill that cannot run because the key is missing, a silent host, and an overdue or unreachable surface. You get another when the backup, host or surface recovers. Each is sent once, not on every check. Alerts lists every event and what each message contains.

Metrics and pings

Set daemon.metrics_addr: "127.0.0.1:9847" and the resident daemon serves Prometheus metrics at http://127.0.0.1:9847/metrics. Each scrape reads the state the daemon wrote after its last backup. Every series except the first two has surface and type labels:

MetricWhat it is
safegrd_daemon_info1, with the daemon's version
safegrd_daemon_state_updated_timestamp_secondsWhen the daemon last wrote its state
safegrd_surface_last_attempt_timestamp_secondsWhen the last backup started
safegrd_surface_last_success_timestamp_secondsWhen the last backup succeeded
safegrd_surface_consecutive_failuresFailed backups since the last success
safegrd_surface_next_due_timestamp_secondsWhen the next scheduled backup is due
safegrd_surface_last_backup_duration_secondsHow long the last successful backup took
safegrd_surface_last_backup_size_bytesWhat it stored, encrypted
safegrd_surface_last_drill_timestamp_secondsWhen the last Fire Drill ran
safegrd_surface_drill_passed1 when the last Fire Drill passed, else 0

A timestamp for something that has never happened is left out instead of written as 0. To alert when the last successful backup is more than a day old:

time() - safegrd_surface_last_success_timestamp_seconds > 86400

Bind it to 127.0.0.1 unless the scraper runs on another machine, because the metrics include your surface names.

To use a check service such as Healthchecks.io or Cronitor, give a surface a ping_url. The daemon posts to URL/start before each backup and to URL after it. If the backup failed, or SafeGrd did not record it, the daemon posts to URL/fail instead. Each post carries one line about what happened as its body.

The daemon pings daemon.ping_url after every pass, so the check alerts you when the daemon stops running. Both settings take env:VAR or file:/path, because anyone who has a check URL can report to that check. If a ping fails, the daemon prints a warning to stderr and the backup's result stands.

daemon:
  metrics_addr: "127.0.0.1:9847"
  ping_url: "env:HEALTHCHECK_DAEMON"
surfaces:
  - id: "orders-db"
    type: "postgres"
    ping_url: "https://hc-ping.com/your-check-uuid"

Fire Drills on an unattended host

When a surface is due for a drill on your plan's schedule, SafeGrd's server tells the daemon. The daemon restores the latest snapshot, checks it against the digests recorded when it was taken, and reports the result to your attestation record.

A drill needs the private key, so the daemon runs one only on a host that already has it (through key_path, SAFEGRD_PRIVATE_KEY, or a SafeGrd-managed key). Every surface in an organization is sealed to that one key, and the daemon never copies it to a host. A host without the key reports Drill blocked: no key, and you are alerted. You can then run safegrd verify wherever you keep the key.

For a PostgreSQL surface, you can give the drill a scratch database. The drill then restores the rows into it and counts them there. Without one, the snapshot is replayed in memory, unless the host can run a throwaway cluster (below).

  - id: "app"
    type: "postgres"
    credential:
      from: env
      name: "APP_DATABASE_URL"
    drill:
      sandbox_url: "env:APP_DRILL_DATABASE_URL"

Create the sandbox once (createdb app_drill) and give the daemon a role that owns it. Use it for nothing else, because a drill restores into it. The daemon refuses a sandbox that holds any table, or one that is the database the surface backs up, and it empties the sandbox after every drill.

If your plan includes sandbox drills, a PostgreSQL surface with no drill.sandbox_url gets a scratch database on the host. When the host has the PostgreSQL server installed (initdb and postgres, of the source database's major version or newer), the daemon creates a throwaway cluster under its state directory and restores the snapshot into it. It deletes the cluster when the drill ends. You do not need Docker.

Where a PostgreSQL surface's drill restores. If drill.sandbox_url is set, into that database, which is emptied after the drill. Otherwise, on a plan without sandbox drills, the snapshot is replayed in memory. On a plan with them, if the host has a PostgreSQL server of the right version and enough disk, into a throwaway cluster deleted when the drill ends; if not, in memory, and the daemon's log says what to install. A drill is due sandbox_url set? in the surface's drill block yes That database emptied after a drill no Sandbox drills? included in your plan no In memory snapshot replayed yes Server on the host? PostgreSQL, initdb, disk yes Throwaway cluster deleted when it ends no In memory the log says what to install
For a PostgreSQL surface. If a drill ran in memory although your plan includes sandbox drills, the console says so under the surface, with the reason.

Look at a failed drill's sandbox

Set drill.keep_failed_sandbox: true to keep the sandbox as it was for 24 hours when a drill restores the snapshot and then fails a check. The daemon prints where the sandbox is, without the password:

Warning: Surface app: sandbox kept at postgres://drill:xxxxx@db.internal/app_drill until 2026-10-06T09:25:25Z; the first drill after that empties it.

A sandbox_url database is left alone for those 24 hours. The next drill is held back, and the first drill after the 24 hours empties it. A throwaway cluster keeps running on its socket until the daemon removes it. Everything stays on the host. If the restore itself fails, it is rolled back and there is nothing to keep. The error is in the daemon's log and in the console.

Install a PostgreSQL server for sandbox drills

On Debian and Ubuntu, install the server with apt install postgresql-16. The postgresql-client package has no server. The server package also starts a server of its own, which the daemon does not use. Stop it with systemctl disable --now postgresql. On macOS, run brew install postgresql@16.

If the daemon cannot start a cluster (no server, a server older than the database, too little disk, or an extension the snapshot uses that the server lacks), the drill runs in memory. The daemon's log says what to install.

When a drill is not the kind your plan includes, the console says why under the surface: it ran in memory because there was no sandbox, or it did not run because the host lacks the disk. A drill that does not run is not a failed backup. The daemon tries again at the next drill, and you are alerted once.

Disk the daemon uses on the host

Database and file backups stream to storage with no copy on the host. Four things do write to the host's disk. Before each one, the daemon checks that it would leave 5% of the filesystem free for everything else on it (at most 1 GiB):

Updates from the console

When a new release is out, the daemon downloads and checks it, and the host's row under Hosts shows ready to install. An owner or admin clicks Update, and the daemon installs the release at its next check-in (every 5 minutes by default), between backups. Before it switches, it checks that:

SafeGrd's server only tells the host which release to install. The host downloads it from the release page (or from daemon.updates_url), so SafeGrd's server never sends you a binary.

The host keeps the release it ran before. If the new one does not check in within two passes, or starts three times without checking in, the host switches back to the earlier one on its own, and the console says why. To go back yourself, click Go back. The earlier release is still on the host, so nothing is downloaded.

Console updates work on hosts set up with install.sh. It keeps each release in /usr/local/lib/safegrd/versions (~/.local/lib/safegrd without sudo) and points /usr/local/bin/safegrd at the one in use. On a host installed before this existed, run the installer once. If you installed with Homebrew, the container image or go install, update the usual way, and the console tells you which command to run. To turn console updates off on a host, set daemon.updates: off.

When SafeGrd's server cannot be reached

The daemon keeps running scheduled backups and writing them to your storage. Each snapshot is in your bucket and restores, and safegrd list shows it.

The daemon prints that it could not send the backup's record, and keeps the record. On the first tick the server answers, the daemon sends it and the console lists the backup. safegrd daemon status shows how many records are waiting.

If the server refuses a report (because the token expired, for example), the daemon does not send it again. It is printed to stderr and shown as not recorded. A backup you run by hand with safegrd backup keeps no state, so its record is not sent again. With --surface, the backup goes through the daemon's state, and its record is.

Three nightly backups while SafeGrd's server is unreachable from Monday 12:00 to Wednesday 12:00. Every backup is written to your bucket on time. Backup 1 is listed in the console at once. Backups 2 and 3 are not listed until the server answers on Wednesday at 12:00, when the daemon sends both records. Your bucket Console Mon 02:00 backup 1 written backup 1 listed Tue 02:00 backup 2 written not listed yet Wed 02:00 backup 3 written not listed yet Mon 12:00 SafeGrd's server unreachable Wed 12:00 server answers 2 and 3 listed
Every backup lands in your bucket on schedule. The daemon keeps the records it could not send and sends them when the server answers. safegrd daemon status shows how many are waiting.