Storage and retention

SafeGrd writes ciphertext into storage you own, or, where the plan includes it, into SafeGrd's own locked bucket (hosted storage). Either way the backup is encrypted on your host first, and who can decrypt it depends on who holds the key (Security and key custody). A bucket of your own has no per-gigabyte surcharge, and it stays in your account.

Sinks

Configure a sink once in config.yaml, or override it per run with --s3-bucket, --s3-endpoint, --s3-region and --s3-prefix. A project can also carry a sink centrally, so a fleet of hosts inherits one destination; the per-run flags win over the project sink, which wins over the file.

Hosted storage

With storage.type: "hosted" the host needs no bucket and never holds a storage credential. For every part it uploads, and every object it reads, it asks the remote server for a URL signed for exactly that request: this part, of this many bytes, or this version of this snapshot. The bytes go straight to SafeGrd's bucket; a URL cannot delete, cannot be used for another object, and expires in minutes. Snapshots there are locked in compliance mode by the server, like in a bucket of your own, and still encrypted on your host.

• Your plan's included storage is counted in locked bytes. A lock cannot be lifted to make room. safegrd doctor shows how much is locked, and the console's Billing tab shows it against your included storage.
• On a paid plan, backups past your included storage are accepted. The server samples what you hold once a day and averages it over the billing period. It charges the part above your included storage per GB-month, rounded down to whole GB, at the end of the period. The price is on the pricing page. You are alerted at 80% and again when you pass your included storage, and the console shows the overage running this period in dollars.
• Locked bytes count until their lock ends. Lowering a retention or a plan does not shorten a lock already written, so overage on those bytes continues until it ends. The console shows the date the last one does.
• On the free plan and during the trial there is no subscription to bill, so a backup that would pass your included storage is refused with the reason, and you are alerted. A subscription that lapses may go a little past its last paid plan's included storage (the margin is in the terms), then backups are refused. Nothing already stored is affected, and restoring always works.
• Retention follows the plan where your config sets none: short locks on every backup, with daily, weekly and monthly copies kept longer, sized so a typical estate fits the plan's included storage. Your own retention_days and keep_* apply within the plan's range: the server sets every lock, and never longer than the plan's longest tier.
• A snapshot is deleted when its lock ends. On hosted storage, retention tracks the Object Lock duration: within 24 hours after a snapshot's lock expires, SafeGrd removes the object from the bucket to free its space. Snapshots remain immutable while their lock is active. Historical records and Fire Drill attestations remain visible in the console, flagged as expired. To retain snapshots indefinitely, export them to your own bucket prior to lock expiration.
• Downloads have a daily limit of four times your included storage, or of what you hold if that is more. It covers every restore and Fire Drill, and a full safegrd export to your own bucket.
• Download a backup from the console. A full backup's details have Download backup, Download metadata and Download RECOVERY.md (the recovery document). The backup arrives encrypted, straight from the bucket. Save the backup and its metadata in one folder and run safegrd restore --from <snapshot id>.safegrd --target <database URL> (or --target-dir for files and mail). If SafeGrd holds the host's key, run it on one of your enrolled hosts: SafeGrd releases the key only to them.
• Incremental snapshots have no backup file to download. Files, PostgreSQL and SQLite back up incrementally unless the surface is set to Full: each run is stored as chunks shared with the other runs of its month. The console offers their metadata and RECOVERY.md. Restore them on a host with safegrd restore --snapshot <snapshot id>, or copy them to your own bucket or a directory with safegrd export. MySQL and MongoDB backups are always full.
• Your owner email must be verified before running your first backup to activate hosted storage.
• Move to your own bucket whenever you like. safegrd export --to-bucket <yours> copies every snapshot across, still encrypted and still locked. Then switch storage.type.

Setting up an AWS S3 bucket

1. Create the bucket with Object Lock enabled. It can only be enabled when the bucket is created; a bucket made without it has to be replaced, not converted.
   aws s3api create-bucket --bucket acme-safegrd --object-lock-enabled-for-bucket
   Outside us-east-1, add --create-bucket-configuration LocationConstraint=eu-west-1 (your region).
2. Create an access key for each backup host with the policy below, and one read-only key for recovery.
3. Run safegrd doctor on the host. It checks that the bucket answers and that Object Lock is on. The first safegrd backup proves the key can write.

What a backup host needs. These are exactly the S3 calls the CLI makes:

That policy cannot delete anything. s3:AbortMultipartUpload only abandons an upload that failed halfway, so its parts are not billed; it cannot touch a finished object. A compromised host cannot even hide a backup behind a delete marker. The only SafeGrd command that deletes is prune, which you opt into (below).

The console's setup wizard writes this policy for your bucket's name, for AWS, Wasabi and MinIO, and the equivalent B2 key command. For AWS it also gives a CloudFormation template and a Terraform file that create the bucket with Object Lock and versioning on, with public access blocked. The same text is at GET /api/v1/storage/bucket-setup?provider=aws&bucket=acme-safegrd.

A recovery machine needs to read and list, nothing more:

The recovery runbooks use exactly this key.

Cloudflare R2

R2 is Cloudflare's S3-compatible store with no egress fees.

1. Create an R2 bucket in the Cloudflare dashboard.
2. Under Manage R2 API Tokens, create an API token with Object Read & Write access for the bucket.
3. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your environment.
4. Point the endpoint at your account URL:

Check connectivity with safegrd doctor before running the first backup.

MinIO (Self-hosted WORM)

MinIO supports real S3 Object Lock when running in multi-drive or distributed erasure-code mode.

1. Create a bucket with object lock enabled using mc:
   mc mb --with-lock myminio/safegrd-backups
2. Set default retention:
   mc retention set --default COMPLIANCE 30d myminio/safegrd-backups
3. Add the sink to config.yaml:

safegrd doctor calls GetBucketObjectLockConfiguration to verify the lock mode.

Backblaze B2

B2 supports Object Lock, but File Lock must be enabled when creating the bucket. It cannot be added later.

1. Create a B2 bucket with File Lock (Object Lock) enabled.
2. In App Keys, generate an Application Key restricted to that bucket with Read and Write permissions.
3. Find your region endpoint (for example, s3.us-east-005.backblazeb2.com).
4. Configure the sink:

Wasabi

Wasabi supports S3 Object Lock in compliance mode with no egress fees.

1. Create a Wasabi bucket with Enable Object Lock turned on in Compliance Mode.
2. Create an Access Key and Secret Key with read and write permissions on that bucket.
3. Set the endpoint for your region:

DigitalOcean Spaces, and any bucket without Object Lock

Spaces has no Object Lock. SafeGrd works there, and the snapshots are still encrypted to your key and still proven by Fire Drills, but nothing stops them being deleted. Anyone holding a key with delete permission on the Space can remove them. Say so in the config, deliberately:

safegrd init writes the same from flags:

The Space's key comes from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the host's environment, or from access_key_id and secret_access_key under storage.

Without worm_mode: "NONE" a backup to Spaces fails at startup rather than pretending to lock. With it, the console marks each snapshot Not locked: deletable, and the CLI never prints an immutability date. Keep the Space's keys few and write-only where you can, and consider copying to a bucket that does lock.

What WORM retention actually means

--retention-days sets an S3 Object Lock retention period on each object as it is written. Under the default, compliance mode, nobody can delete or overwrite the object until retention expires: not the account's root user, and not SafeGrd. Stolen administrator credentials cannot remove a locked backup.

worm_mode: "GOVERNANCE" is the softer variant, where a principal holding the bypass permission can shorten a lock. It protects against accidents, but an attacker with that permission can remove the backup.

Keeping daily, weekly and monthly backups longer

The daemon can keep some backups longer than others, grandfather-father-son style, set per surface or under defaults:

• The tier is decided when the backup is written, because an Object Lock period can never be shortened later. The first successful backup in a day, week or month takes that slot. On an hourly schedule this is what keeps storage small: the hourly copies need only a short lock, and one backup a day carries the history. A failed one does not, so the next success takes it instead. The daemon logs which backup took a slot.
• If the daemon loses its state (a new host, a wiped state directory), the next backup takes the slots again. That keeps one backup longer than planned, never shorter.
• A locked backup is still never deleted by SafeGrd. The tiers decide how long each one is protected. A lifecycle rule on the bucket decides when it goes (see Listing and expiry below).
• Only the daemon applies tiers. A one-off safegrd backup uses --retention-days.
• Under worm_mode: "NONE" nothing is locked, so the tiers protect nothing and the daemon does not claim them.

Sinks with no Object Lock

worm_mode: "NONE" is for storage with no Object Lock, such as DigitalOcean Spaces. SafeGrd never falls back to unlocked storage on its own: a bucket without Object Lock and without this setting fails at startup. With NONE, snapshots are written unlocked. The value is case-sensitive: COMPLIANCE, GOVERNANCE or NONE.

Listing and expiry

In your own bucket, SafeGrd deletes nothing unless you turn pruning on. (Hosted storage prunes a snapshot once its lock ends.) An expired lock only allows deletion. The snapshot stays until something prunes it.

To let expired snapshots go, set storage.expire_after_lock: true and the daemon prunes once a day, or run safegrd prune (with --dry-run first). It deletes a snapshot only after its Object Lock ended more than a day ago, judged by the bucket's own clock and the lock the bucket itself reports, rather than the host's clock. It keeps the newest three snapshots of each surface whatever their locks say (--min-keep), so a short retention never empties a surface, and retains the last known-good snapshot while Threat Shield reports an open anomaly. If the remote server is unreachable, nothing is pruned. Pruning requires s3:DeleteObjectVersion, s3:GetObjectRetention and s3:GetObjectLegalHold on the host key (the wizard adds these when expiry is enabled). With them the key can delete expired snapshots, and still cannot touch a locked one. Use this rather than a bucket lifecycle rule, which cannot tell the daily, weekly and monthly tiers apart. Local sinks are never pruned.

Keep one snapshot longer

A lock can be extended, which is the one change Object Lock allows. Keep a snapshot longer before a migration, or because an auditor asked for it, up to 13 months from today. Nobody can shorten the lock afterwards, SafeGrd included.

• Hosted storage: an owner or admin picks the date under the snapshot's Details in the console. The kept snapshot counts against the hosted storage it uses for as long as it is locked.
• Your own bucket: run safegrd keep --snapshot <id> --until 2027-01-31 --yes on a host that writes to it. The host's key needs s3:PutObjectRetention and s3:GetObjectRetention. The console shows the new date once the host reports it.

The backup file, its metadata and its recovery document are extended together. An incremental snapshot shares its chunks with the rest of its month, so it is kept as long as that month and not on its own.

When a project changes buckets

Moving a project to another bucket, or to or from hosted storage, or deleting an organization, leaves the backups already written where they are. SafeGrd writes down what stays: the key access log gets a sink_released entry naming every snapshot it recorded in the place being left, and the console says the same when you make the change. In your own bucket, your lifecycle rule or safegrd prune removes them once their locks end. Hosted storage removes each one when its lock ends.

A recovery document beside every snapshot

Each snapshot is written with two files beside it: SNAPSHOT.meta.json, the manifest, and SNAPSHOT.RECOVERY.md, a plain-text page that says where the snapshot is (bucket, endpoint, prefix), what a restore needs (the server version, the extensions and the roles to have ready), and the commands that restore it, with safegrd restore and, for PostgreSQL, with psql alone. It is written for a reader who has the bucket and the key and nothing else, so it is not encrypted and it holds no secret: no credential, no key, no connection string.

A full backup's two files sit next to its archive. An incremental snapshot's sit in the snapshots/ folder of its month's repository, repo/<surface>/<epoch>/snapshots/. On hosted storage, download them from the snapshot's details in the console.

recovery_doc_sealed: true on a surface seals it with the surface's recipient instead, as SNAPSHOT.RECOVERY.md.age. Both are locked like the manifest. A recovery document that could not be written is said on stderr; the backup still succeeds.

Someone deleted my backups

In a versioned bucket with Object Lock on, an ordinary delete cannot remove a locked object. S3 adds a delete marker on top of it instead, which hides the snapshot from normal listings and leaves its data where it was.

safegrd undelete --all rolls a versioned bucket back to before the deletion: every snapshot and sidecar is visible again once its delete marker is gone, and nothing is re-uploaded, so the rollback takes seconds whatever the bucket holds.

The command only ever removes delete markers, addressed by version id. It cannot remove a version holding data, which is why it is safe to run twice. If it finds nothing to remove it says which of two reasons applies: the snapshot is there and was never hidden, or the id is not in that bucket at all. In the second case it names the bucket, prefix and node it searched and exits non-zero. Removing a delete marker needs s3:DeleteObjectVersion, which a backup host's key does not have, so run it with an administrator's key. Afterwards, deny s3:DeleteObject to the credential that did it.