Docs / Storage and retention

Storage and retention

Your backups go to a bucket you own or, if your plan includes it, to SafeGrd's own locked bucket (hosted storage). Either way, each backup is encrypted on your host before it leaves. Who can decrypt it depends on how your key is managed (Security and key custody). A bucket of your own has no per-gigabyte surcharge, and it stays in your account.

Choose a sink

SinkNotes
hostedSafeGrd's own bucket, locked in compliance mode, with your plan's included storage. You have no bucket to set up. See Hosted storage.
localA directory on the host. A filesystem-level lock stands in for Object Lock. Use it to try SafeGrd or on an air-gapped host. If the machine is lost, so are the backups.
AWS S3The reference target. Nobody can shorten or remove a compliance-mode lock on AWS, and SafeGrd's retention depends on that.
MinIOSelf-hosted and S3-compatible, with real Object Lock. Set endpoint.
Cloudflare R2S3-compatible. Set endpoint. It has no S3 Object Lock, so set worm_mode: "NONE" (details).
Backblaze B2, WasabiS3-compatible, each with its own object lock features. Check the bucket with safegrd doctor.
DigitalOcean SpacesS3-compatible, but it has no Object Lock at all. You have to opt out of locking explicitly (details).
SFTPA directory on an SSH server. It has no Object Lock, and it accepts worm_mode: "NONE" and nothing else (details).

Set the sink once in config.yaml, or override it for one run with --s3-bucket, --s3-endpoint, --s3-region and --s3-prefix. You can also set a sink on a project, and every host in the project uses it. When more than one is set, the per-run flags win over the project's sink, and the project's sink wins over config.yaml.

Choosing a sink. Just trying SafeGrd, or a host with no network: local, a directory lost with the machine. No bucket of your own: hosted storage, with nothing to set up. A bucket of your own with S3 Object Lock in compliance mode, such as AWS S3, MinIO, Backblaze B2 or Wasabi: s3, locked. A bucket with no Object Lock, such as Cloudflare R2, DigitalOcean Spaces or an SFTP server: worm_mode NONE. Just trying it? or a host with no network yes local lost with the machine no A bucket of your own? in your cloud account no hosted no bucket to set up yes Does it lock? S3 Object Lock, compliance yes s3, locked S3, MinIO, B2, Wasabi no worm_mode: "NONE" R2, Spaces, SFTP
On B2 and Wasabi, safegrd doctor confirms the bucket locks. With worm_mode: "NONE", anyone with a delete key on the bucket can remove the snapshots.

Hosted storage

Set storage.type: "hosted" and you have no bucket to set up. The host never holds a storage credential. Each time it uploads a part or reads an object, it asks SafeGrd's server for a URL signed for that one request: this part, of this many bytes, or this version of this snapshot. The bytes go straight to SafeGrd's bucket. A signed URL cannot delete, cannot be used for another object, and expires in minutes. The server locks each snapshot in compliance mode, as in a bucket of your own, and the data is still encrypted on your host.

safegrd init --storage hosted --database-url env:DATABASE_URL
safegrd enroll --token <your access token>
safegrd backup

Hosted storage is activated once your owner email is verified, so verify it before you run the first backup.

Included storage and overage

Your plan's included storage is counted in locked bytes. On the free plan, every byte kept counts, locked or not. You cannot lift a lock to make room. safegrd doctor shows how much is locked, and the console's Billing tab shows it against your included storage.

On a paid plan, a backup that goes past your included storage is still accepted. Once a day the server samples what you hold, and it averages the samples over the billing period. At the end of the period it charges for the part above your included storage, per GB-month, rounded down to whole GB. The price is on the pricing page. You get an alert at 80% and another when you pass your included storage. The console shows this period's overage so far, in dollars.

Locked bytes count until their lock ends. If you lower a retention or move to a smaller plan, the locks already written keep their dates, and overage on those bytes continues until each lock ends. The console shows the date the last one ends.

On the free plan and during the trial, there is no subscription to bill. A backup that would pass your included storage is refused with the reason, and you get an alert. If a subscription lapses, you can go a little past the last paid plan's included storage (the terms give the margin). After that, backups are refused. Nothing already stored is affected, and you can always restore.

How long hosted backups are kept

If your config sets no retention, the plan's applies. Every backup gets a short lock, and daily, weekly and monthly copies are kept longer. The lengths are sized so a typical estate fits in the plan's included storage. You can set your own retention_days and keep_* within the plan's range. The server sets every lock, and never longer than the plan's longest tier.

On the free plan, a backup is kept for 1 to 7 days (7 unless you choose otherwise) and locked for its first day. After that day it stays in the bucket unlocked until its retention ends, and it counts against your included storage until it is deleted. Each backup is a full copy, so each one can be deleted on its own date.

A snapshot is deleted when its retention ends, which on a paid plan is when its lock expires. SafeGrd removes the object from the bucket within 24 hours after that. While a snapshot is locked, nobody can change or delete it. Its verification records and Fire Drill certificates stay in the console, marked expired. To keep snapshots longer, export them to your own bucket before the lock expires.

Download a backup

You can download up to four times your included storage a day, or as much as you hold if that is more. The limit covers every restore and Fire Drill, and a full safegrd export to your own bucket.

To download a full backup, open its details in the console. Click Download backup, Download metadata or Download RECOVERY.md (the recovery document). The backup arrives encrypted, straight from the bucket. Save the backup and its metadata in one folder and run safegrd restore --from <snapshot id>.safegrd --target <database URL> (or --target-dir for files and mail). If the host has a SafeGrd-managed key, run the restore on one of your enrolled hosts, because SafeGrd releases the key only to them.

An incremental snapshot has no backup file to download. Files, PostgreSQL and SQLite back up incrementally unless the surface is set to Full. Each run is stored as chunks it shares with the other runs of its month. For these, the console offers the metadata and RECOVERY.md. Restore one on a host with safegrd restore --snapshot <snapshot id>, or copy it to your own bucket or a directory with safegrd export. MySQL and MongoDB backups are always full.

Move to your own bucket

You can move whenever you like. safegrd export --to-bucket <yours> copies every snapshot across, still encrypted and still locked. Then switch storage.type.

WordPress sites and databases SafeGrd backs up, in your own bucket

A WordPress site, or a PostgreSQL database that SafeGrd backs up without a host of yours, can back up to the project's own bucket. SafeGrd holds that bucket's key and signs each upload and read with it, the same way it signs them for hosted storage with its own key. Neither the site nor the database backup is ever given the key. All of these have to be true:

Until then, a site backs up to hosted storage. The surface's panel in the console, the plugin's Tools page and wp safegrd status tell you which condition is missing. You can add a database SafeGrd backs up to such a project only once all of them are true, and only as an incremental backup. A full backup is one archive, and SafeGrd writes those only to hosted storage. MySQL and MongoDB are added to a project on hosted storage.

In your bucket, each backup is locked for the project's retention, in the bucket's lock mode. In a bucket without Object Lock, a site's backups are not locked, and the plugin and the console show them as Not locked. SafeGrd reads backups by name, so the bucket does not need versioning.

Hosted storage's included amount, plan retention and download limit do not apply in your bucket. SafeGrd deletes nothing there. Your bucket's lifecycle rule does, or safegrd prune on a host with the key.

The backups are under <prefix>/<node id>/repo/, the same layout a host writes. An enrolled host with the bucket's key can run safegrd verify and safegrd restore on them straight from the bucket. When the project moves between hosted storage and its bucket, the next backup uploads everything once. Older backups stay where they were written.

Storage forecast

Open Projects in the console and click Storage on a project. It shows what the project's storage holds now, and what it will hold once every surface has run through its retention. On hosted storage, it also shows the date you will reach your plan's included storage.

The forecast replays the last 30 days. It assumes each backup written in that time is written again 30 days later, at the same size and with the same lock. What is already stored drops out on the day its lock ends. A surface that has backed up for less than 30 days is replayed over the time it has run. Because it replays what your hosts wrote, the forecast includes compression, the longer locks on weekly and monthly copies, and an incremental surface's larger first run each month. It does not predict growth. When a database grows, its backups grow, and the forecast follows.

A surface with no backup yet is listed but not counted. On a bucket without Object Lock, a surface with no retention set is not counted either, because nothing removes its backups. To bring the figure down, shorten retention, back up less often, or move a surface to another project's storage. For your own bucket, the Object Lock cost calculator turns a schedule and retention into a monthly bill.

Setting up an AWS S3 bucket

  1. Create the bucket with Object Lock enabled. You can only turn it on when you create the bucket. A bucket made without it cannot be converted, so you would have to replace it.
    aws s3api create-bucket --bucket acme-safegrd --object-lock-enabled-for-bucket
    Outside us-east-1, add --create-bucket-configuration LocationConstraint=eu-west-1 (your region).
  2. Create an access key with the policy below for each backup host, and one read-only key for recovery.
  3. Run safegrd doctor on the host. It checks that the bucket answers and that Object Lock is on. The first safegrd backup proves the key can write.

What a backup host needs. These are the S3 calls the CLI makes, and no others:

{"Version": "2012-10-17", "Statement": [
  {"Effect": "Allow", "Action": ["s3:ListBucket", "s3:ListBucketVersions", "s3:GetBucketObjectLockConfiguration"],
   "Resource": "arn:aws:s3:::acme-safegrd"},
  {"Effect": "Allow", "Action": ["s3:PutObject", "s3:PutObjectRetention", "s3:AbortMultipartUpload", "s3:GetObject", "s3:GetObjectVersion"],
   "Resource": "arn:aws:s3:::acme-safegrd/*"}
]}

That policy cannot delete anything. s3:AbortMultipartUpload only abandons an upload that failed halfway, so you are not billed for its parts. It cannot touch a finished object. A compromised host cannot hide a backup behind a delete marker either. The only SafeGrd command that deletes is prune, and it is off until you turn it on (below).

The console's setup wizard writes this policy with your bucket's name for AWS, Wasabi and MinIO, and the matching key command for B2. For AWS it also gives you a CloudFormation template and a Terraform file. Both create the bucket with Object Lock and versioning on and public access blocked. The API returns the same text at GET /api/v1/storage/bucket-setup?provider=aws&bucket=acme-safegrd.

A recovery machine needs only to read and list:

{"Version": "2012-10-17", "Statement": [
  {"Effect": "Allow", "Action": ["s3:ListBucket", "s3:ListBucketVersions"], "Resource": "arn:aws:s3:::acme-safegrd"},
  {"Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectVersion"], "Resource": "arn:aws:s3:::acme-safegrd/*"}
]}

The recovery runbooks use this key.

A colder storage class

Set storage.storage_class to choose the S3 storage class of each archive: STANDARD (the default), STANDARD_IA or GLACIER_IR. A colder class costs less to hold but charges for each GB read, and every Fire Drill reads one copy. It also has a minimum billed period (30 and 90 days on AWS), and you pay for that period even when the lock is shorter. Price your schedule and drills in the Object Lock cost calculator before you change the class.

The manifest beside each archive stays STANDARD, because every list and drill reads it. Incremental (format: repo) surfaces are not affected. Glacier Flexible Retrieval and Deep Archive are refused, because a drill could not restore from them.

Cloudflare R2

R2 is Cloudflare's S3-compatible store, with no egress fees. It has no S3 Object Lock API. It refuses PutObjectLockConfiguration, so SafeGrd cannot lock a backup there or read a lock back. Set worm_mode: "NONE", as for any bucket without Object Lock.

  1. Create an R2 bucket in the Cloudflare dashboard.
  2. Under Manage R2 API Tokens, create an API token with Object Read & Write access for this bucket only. R2 tokens cannot be narrowed further, so this one can also delete.
  3. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your environment.
  4. Point the endpoint at your account URL:
storage:
  type: "s3"
  bucket: "acme-safegrd"
  region: "auto"
  endpoint: "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"
  worm_mode: "NONE"

R2 has bucket locks of its own. A bucket lock keeps objects from being deleted or overwritten for a period you set on the bucket, from the dashboard or with npx wrangler r2 bucket lock add <bucket>. If you want one, make it at least as long as retention_days. SafeGrd cannot see the rule, so the console still marks each snapshot Not locked: the bucket has no Object Lock. Anyone with permission on the Cloudflare account can remove the rule.

Run safegrd doctor before the first backup to check the connection.

MinIO (self-hosted WORM)

MinIO has real S3 Object Lock when it runs in multi-drive or distributed erasure-code mode.

  1. Create a bucket with Object Lock on, using mc:
    mc mb --with-lock myminio/safegrd-backups
  2. Set a default retention:
    mc retention set --default COMPLIANCE 30d myminio/safegrd-backups
  3. Save the AWS backup policy above, with your bucket's name, as safegrd-backup.json, and give the backup hosts a user that has only it:
    mc admin policy create myminio safegrd-backup safegrd-backup.json
    mc admin user add myminio safegrd-backup CHOOSE-A-SECRET-KEY
    mc admin policy attach myminio safegrd-backup --user safegrd-backup
  4. Add the sink to config.yaml:
storage:
  type: "s3"
  bucket: "safegrd-backups"
  region: "us-east-1"
  endpoint: "http://minio.internal:9000"
  worm_mode: "COMPLIANCE"

safegrd doctor calls GetBucketObjectLockConfiguration to check the lock mode.

Backblaze B2

B2 has Object Lock, which it calls File Lock. Turn it on when you create the bucket, because it cannot be added later.

  1. Create a B2 bucket with File Lock (Object Lock) enabled.
  2. Create an application key limited to that bucket, with only the capabilities a backup needs. Without deleteFiles, the key cannot delete or hide a file:
    b2 key create --bucket acme-b2-safegrd safegrd-backup listBuckets,readBuckets,readBucketRetentions,listFiles,readFiles,writeFiles,writeFileRetentions
  3. Find your region endpoint (for example, s3.us-east-005.backblazeb2.com).
  4. Configure the sink:
storage:
  type: "s3"
  bucket: "acme-b2-safegrd"
  region: "us-east-005"
  endpoint: "https://s3.us-east-005.backblazeb2.com"
  worm_mode: "COMPLIANCE"

Wasabi

Wasabi has S3 Object Lock in compliance mode, and no egress fees.

  1. Create a Wasabi bucket with Enable Object Lock turned on in Compliance Mode.
  2. Under Policies, create a policy with the AWS backup policy above, with your bucket's name. Wasabi reads the same document. Create a user for the backup hosts, attach the policy, and make an access key.
  3. Set the endpoint for your region:
storage:
  type: "s3"
  bucket: "acme-wasabi-safegrd"
  region: "eu-central-1"
  endpoint: "https://s3.eu-central-1.wasabisys.com"
  worm_mode: "COMPLIANCE"

DigitalOcean Spaces, and any bucket without Object Lock

Spaces has no Object Lock. SafeGrd works there: snapshots are encrypted to your key, and Fire Drills test them as usual. But nothing stops them from being deleted. Anyone with a key that has delete permission on the Space can remove them. To use a Space, set worm_mode: "NONE" in the config:

storage:
  type: "s3"
  bucket: "acme-safegrd"
  region: "nyc3"
  endpoint: "https://nyc3.digitaloceanspaces.com"
  worm_mode: "NONE"

safegrd init writes the same from flags:

safegrd init --storage s3 --s3-bucket acme-safegrd --s3-region nyc3 --s3-endpoint https://nyc3.digitaloceanspaces.com --worm-mode NONE

The Space's key comes from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the host's environment, or from access_key_id and secret_access_key under storage.

Without worm_mode: "NONE", a backup to Spaces fails at startup, so an unlocked backup is never reported as locked. With it, the console marks each snapshot Not locked: the bucket has no Object Lock, and the CLI never prints an immutability date. safegrd doctor reports it as a warning: worm_mode is NONE: snapshots in this bucket can be deleted by anyone with delete permission. On a bucket that should lock and does not, safegrd doctor fails with bucket ... cannot be confirmed to lock and tells you what to change.

Hand out as few keys to the Space as you can, write-only where possible. Consider also copying to a bucket that does lock.

An SFTP server

You can back up to a directory on any SSH server that offers SFTP, such as a NAS, a storage box or a machine of your own. Every surface type works there, in both formats (incremental and format: tar). Restores and Fire Drills read from it the same way as from a bucket.

An SFTP server has no Object Lock, so set worm_mode: "NONE". A config with any other value, or with none, is refused when it is read. Each snapshot records the date it is kept until, and the CLI never prints a lock date for it.

storage:
  type: "sftp"
  worm_mode: "NONE"
  retention_days: 30
  sftp:
    host: "backup.example.com"
    port: 22
    user: "safegrd"
    path: "backups"
    private_key: "file:/etc/safegrd/sftp_key"
    known_hosts: "/etc/safegrd/known_hosts"

path is the directory backups go in, either absolute or relative to the account's login directory. Sign in with private_key (a key without a passphrase) or password. Both take env:VAR or file:/path, so the secret stays out of the config.

The CLI checks the server's host key before it sends anything. You can list the server in an OpenSSH known_hosts file: run ssh-keyscan -p 22 backup.example.com >> /etc/safegrd/known_hosts, then compare the fingerprints with the server's administrator. Or pin the key's fingerprint in place of known_hosts:

# on the server
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
# in the storage block, under sftp:
host_key: "SHA256:..."

If the server's key does not match, or known_hosts does not list the server, the CLI refuses to connect, and the error shows the fingerprint the server presented. Give known_hosts as a full path, because the daemon runs as its own user, and that user's home is not yours.

Give the account write access to path and nothing else. Anyone who can sign in as that account can delete the backups in it, so keep its key or password on the host that backs up.

safegrd init does not write an SFTP block. Run it with --storage local, then replace the storage block with the one above. safegrd doctor signs in, lists the directory and reports that the server has no Object Lock. WAL archiving for point-in-time recovery does not use SFTP: it needs a bucket, hosted storage or a local directory.

How WORM retention works

--retention-days sets an S3 Object Lock retention period on each object as it is written. In compliance mode, the default, nobody can delete or overwrite the object until retention expires. That includes the account's root user and SafeGrd. Stolen administrator credentials cannot remove a locked backup either.

You cannot shorten or undo a lock.

Once a compliance lock is set, nobody can shorten or cancel it. With a long retention on hourly snapshots, you pay for storage you cannot delete until each date passes. Try a short retention_days on a small dataset first.

worm_mode: "GOVERNANCE" is the softer mode. Anyone holding the bypass permission can shorten a lock. It protects against accidents, but an attacker with that permission can remove the backup.

Keeping daily, weekly and monthly backups longer

You can keep some backups longer than others (grandfather-father-son). Set the tiers on a surface or under defaults:

retention_days: 2 # every backup
keep_daily: 14 # the first backup of each UTC day, for 14 days
keep_weekly: 4 # the first backup of each ISO week, for 4 weeks
keep_monthly: 12 # the first backup of each month, for 12 months
Lock lengths for the example config: every backup 2 days, the first of each day 14 days, the first of each week 4 weeks, the first of each month 12 months. every backup retention_days: 2 2 days first of each day keep_daily: 14 14 days first of each week keep_weekly: 4 4 weeks first of each month keep_monthly: 12 12 months
How long each backup is locked with the settings above. The bars are not to scale.

Locking only the kept copies

When you create a project, you choose which of its backups are locked. The organization's first project locks every backup. A project you create after that can lock only the copies it keeps:

Sinks with no Object Lock

Use worm_mode: "NONE" for storage with no Object Lock, such as DigitalOcean Spaces. SafeGrd never switches to unlocked storage on its own. A backup to a bucket without Object Lock fails at startup unless this setting is there. With NONE, snapshots are written unlocked. The value is case-sensitive: COMPLIANCE, GOVERNANCE or NONE.

Listing and expiry

# what is in the bucket, and until when it is locked
safegrd list
# sink reachability, lock configuration, key presence
safegrd doctor

In your own bucket, SafeGrd deletes nothing until you turn pruning on. (Hosted storage prunes a snapshot once its retention ends.) When a lock expires, the snapshot may be deleted, but it stays until something prunes it. With pruning off, the bucket and its bill grow with every backup. A surface's panel in the console shows what it keeps now and what it will keep once its copies have rolled over. How fast it grows is in the storage forecast.

A local directory is never pruned. Once you no longer need its snapshot files, anyone who can write to the directory can delete them.

To let expired snapshots go, set storage.expire_after_lock: true and the daemon prunes once a day. Or run safegrd prune yourself (with --dry-run first).

Pruning deletes a snapshot only after its Object Lock ended more than a day ago. It judges that by the bucket's own clock and the lock the bucket itself reports, not by the host's clock. It keeps the newest three snapshots of each surface whatever their locks say (--min-keep), so a short retention never empties a surface. It also keeps the last known-good snapshot while Threat Shield reports an open anomaly. If SafeGrd's server is unreachable, nothing is pruned.

Pruning needs s3:DeleteObjectVersion, s3:GetObjectRetention and s3:GetObjectLegalHold on the host's key. The wizard adds these when you turn on expiry. With them, the key can delete any snapshot whose lock has ended, and it still cannot touch a locked one. The bucket enforces the lock. min_keep and the anomaly hold are enforced by safegrd itself, so set retention_days to the furthest back you would need to restore. Neither the backup policy nor the recovery policy grants s3:DeleteObject, so a host's key cannot write a delete marker over a snapshot.

Use pruning instead of a bucket lifecycle rule, because a lifecycle rule cannot tell the daily, weekly and monthly tiers apart.

One snapshot with retention_days 14. It is locked for 14 days. With expire_after_lock true, it becomes deletable when the lock ends and is pruned once the lock has been over for a day. With pruning off, the default, it stays in the bucket after the lock ends and is billed until you delete it. expire_after_lock: true locked 1 day pruned Pruning off, the default locked kept, and billed backup written lock ends, day 14
retention_days: 14 on one snapshot. The daemon prunes once a day, and keeps the newest three snapshots of each surface whatever their dates.

An SFTP server has no lock. There, safegrd prune and expire_after_lock delete a snapshot one day after the keep-until date it records. The day is judged by the server's clock: the time it stamps on a file it has just written. The newest min_keep snapshots and the anomaly hold apply there too.

The Snapshots tab: each snapshot with its surface, contents, size, Threat Shield state and the date it is locked until. The Snapshots tab: each snapshot with its surface, contents, size, Threat Shield state and the date it is locked until.
The console's Snapshots tab shows what each snapshot holds and the date its lock runs to.

Keep one snapshot longer

You can extend a snapshot's lock up to 13 months from today, before a migration or because an auditor asked for it. Extending is the one change Object Lock allows. Nobody can shorten the lock afterwards, SafeGrd included.

On hosted storage, an owner or admin picks the date under the snapshot's Details in the console. The kept snapshot counts against your hosted storage for as long as it is locked.

In your own bucket, run safegrd keep --snapshot <id> --until 2027-01-31 --yes on a host that writes to it. The host's key needs s3:PutObjectRetention and s3:GetObjectRetention. The console shows the new date once the host reports it.

The backup file, its metadata and its recovery document are extended together. An incremental snapshot shares its chunks with the rest of its month, so it is kept as long as that month and cannot be kept on its own.

When a project changes buckets

If you move a project to another bucket, or to or from hosted storage, or delete an organization, the backups already written stay where they are. SafeGrd records what was left behind. The key access log gets a sink_released entry naming every snapshot SafeGrd recorded in the old location, and the console tells you the same when you make the change. In your own bucket, your lifecycle rule or safegrd prune removes them once their locks end. Hosted storage removes each one when its lock ends.

A second bucket

You can have every backup written to up to two more S3 buckets, in another account, at another provider or in another region. Each bucket has its own access key, lock and retention. Every copy is encrypted to the same Age key as the first, so the key that restores one restores them all.

Add a bucket in the console

On Growth and Scale:

  1. Create the bucket at your provider. Turn on Object Lock if you want its copies locked. Make an access key that can write to it.
  2. Open Projects, click Storage on the project, and go to More buckets.
  3. Click Add bucket. Give it a name, such as offsite, and fill in the bucket, endpoint, region and keys.
  4. Choose its lock and how many days to keep each backup. Click Test, then Save.

Every host in the project writes to the bucket from its next backup. SafeGrd holds the bucket's key, encrypted, and gives it only to your enrolled hosts.

Or list it in the host's config

This works on every plan. Add the buckets under copies: in the host's config:

copies:
- name: offsite
type: s3
bucket: company-backups-offsite
endpoint: https://s3.us-west-004.backblazeb2.com
region: us-west-004
worm_mode: COMPLIANCE
retention_days: 90
access_key_id: env:B2_KEY_ID
secret_access_key: env:B2_KEY

Every surface on the host writes to every bucket in the list. To choose for one surface, set copies: [offsite] on it, or copies: [] for none. If the project has buckets set in the console, those replace this list, and the host says so on each run. Every key is listed in the config reference.

See what each bucket holds

shop-db's side panel: under Copies, the offsite bucket holds 2 backups, locked until 30 Jan, not drilled from yet. shop-db's side panel: under Copies, the offsite bucket holds 2 backups, locked until 30 Jan, not drilled from yet.
A surface's copies: how many backups each bucket holds, its lock, and when a drill last restored from it.

Each snapshot in the surface's Snapshots tab says how many copies it has, such as "2 of 2 copies". If a bucket is behind, the surface shows Copy behind. On the host, safegrd copies status lists what each bucket lacks.

How a backup reaches each bucket

When a bucket is down

If one bucket is down, the backup still succeeds on your other buckets. The host prints Warning: copy to offsite failed, and the surface shows Copy behind. The daemon tries every half hour, and once the bucket answers, it writes what the bucket lacks. After a day behind, you get a copy_behind alert, and a copy_caught_up alert when the bucket holds every backup again. To catch up now, run:

safegrd copies sync

Restore from a copy

A copy holds the same files as your own bucket, so you restore from it the same way:

safegrd restore --snapshot snap-1b094c0af8 --from-bucket offsite --target "$DATABASE_URL"

verify, list and find take --from-bucket too. Without SafeGrd, you restore from a copy the same way as from your own bucket (runbooks). To test that each copy restores, set drill.source: rotate on the surface. Each Fire Drill then reads the next bucket, and its certificate names the bucket it read.

Limits

A recovery document beside every snapshot

Every snapshot is written with two files beside it: SNAPSHOT.meta.json (the manifest) and SNAPSHOT.RECOVERY.md, a plain-text recovery page. The page says:

It is written for someone who has the bucket and the key and nothing else. So it is not encrypted, and it holds no secret: no credential, no key and no connection string.

For a full backup, the two files sit next to its archive. For an incremental snapshot, they sit in the snapshots/ folder of its month's repository, repo/<surface>/<epoch>/snapshots/. On hosted storage, download them from the snapshot's details in the console.

Set recovery_doc_sealed: true on a surface to seal the recovery page with the surface's recipient instead, as SNAPSHOT.RECOVERY.md.age. Both forms are locked like the manifest. If the recovery document cannot be written, the CLI says so on stderr, and the backup still succeeds.

Someone deleted my backups

You can bring them back with safegrd undelete. In a versioned bucket with Object Lock on, an ordinary delete cannot remove a locked object. S3 puts a delete marker on top of it instead. The marker hides the snapshot from normal listings, and the data stays where it was.

# make one snapshot visible again
safegrd undelete --snapshot snap-1b094c0af8
# clear every delete marker in the sink
safegrd undelete --all

safegrd undelete --all rolls a versioned bucket back to before the deletion. Every snapshot and sidecar is visible again once its delete marker is gone. Nothing is re-uploaded, so the rollback takes seconds however much the bucket holds.

The command removes only delete markers, each addressed by its version id. It cannot remove a version that holds data, so it is safe to run twice. If it finds nothing to remove, it tells you why: either the snapshot is there and was never hidden, or the id is not in that bucket at all. In the second case it names the bucket, prefix and node it searched, and exits non-zero.

Removing a delete marker needs s3:DeleteObjectVersion. A host's key has that only when expiry is on, so run the command with an administrator's key. Afterwards, deny s3:DeleteObject to the credential that made the deletion.