Alerts
SafeGrd tells you when a backup or a drill needs your attention. Alerts come from SafeGrd's server rather than your hosts, so you still hear about a host that has gone offline.
Where alerts go
| Channel | Plans | How |
|---|---|---|
| The console | Every plan | Nothing to set up. Alerts in the
console's top bar lists every alert from the last 30 days, newest first, with the surface
it is about. Every member of the organization sees it, viewers included. An access token
can read it from GET /api/v1/orgs/{id}/events. |
| Webhook | Every plan, Free included | Paste a URL under Settings → Alerts in the console, then press Send a test. Slack incoming-webhook URLs get Slack's message format, Discord webhook URLs get Discord's, and Teams workflow URLs get an Adaptive Card. Any other URL gets the JSON below. Getting the URL says where Slack, Discord and Teams show it. |
| PagerDuty | Every plan, Free included | Paste an Events API v2 integration key under Settings → Alerts. A failure or a warning opens an incident on that service, and its recovery resolves it. You get one incident per surface and condition. A notice (storage at 80%, a bucket caught up) opens no incident. Getting the key says where PagerDuty shows it. What fires lists which alerts resolve on their own. |
| Paid plans, off until you turn it on | An owner or admin turns on Email alerts to owners and admins under Settings → Alerts. Alerts then go to every owner and admin of the organization, at the address they sign in with. Email is there to send you to the console, so it is capped: at most one email per kind of alert for a surface in a day, and one per kind across the organization in an hour. The webhook and the console get every event. |
Hosts do not send alerts, so a host's config.yaml has no alert settings. If
safegrd doctor finds an alert: block, it says so.
The webhook must be a public address. SafeGrd refuses a URL that points at a private or loopback network, both when you save it and again when it connects. It does not follow redirects.
Getting the URL
Discord
- In the Discord server, open Server Settings → Integrations → Webhooks. You need the Manage Webhooks permission on the server.
- Press New Webhook, name it (for example SafeGrd) and choose the channel alerts go to.
- Press Copy Webhook URL. It looks like
https://discord.com/api/webhooks/1234567890/AbC.... - In the SafeGrd console, paste it under Settings → Alerts, press Save, then Send a test. The test message shows up in that channel.
Slack
- Go to api.slack.com/apps, press Create New App, choose From scratch and pick your workspace.
- Under Incoming Webhooks, turn on Activate Incoming Webhooks.
- Press Add New Webhook to Workspace and choose the channel alerts go to. If your workspace limits app installs, an admin has to approve it first.
- Copy the URL Slack shows. It starts with
https://hooks.slack.com/services/. - Paste it under Settings → Alerts, press Save, then Send a test.
Microsoft Teams
- In Teams, open the channel, press More options (…) and choose Workflows.
- Pick the template Send webhook alerts to a channel (its trigger is When a Teams webhook request is received), choose the team and channel, and press Add workflow.
- Copy the URL the workflow shows. It is on
logic.azure.com,powerplatform.comorpowerautomate.com, with asig=parameter. - Paste it under Settings → Alerts, press Save, then Send a test. The test shows up in that channel as a card.
Microsoft retired the Office 365 connector webhooks (webhook.office.com) in May 2026.
If one of those URLs is still pasted, it gets the same card. A workflow belongs to the person who
made it, so add a co-owner in Power Automate to keep it running after they leave.
Paste the URL exactly as Slack, Discord or Teams gives it. SafeGrd picks the message format from
the address: hooks.slack.com for Slack, discord.com/api/webhooks/ for
Discord, and logic.azure.com, powerplatform.com or
powerautomate.com for Teams. A URL behind a shortener or proxy gets the generic
JSON, which Slack, Discord and Teams reject.
Anyone with the URL can post to that channel, so keep it out of tickets and chat. To replace it, delete the webhook in Slack, Discord or Teams, make a new one and paste that.
PagerDuty
- In PagerDuty, open the service alerts should page for and its Integrations tab.
- Press Add an integration, choose Events API V2 and press Add.
- Copy the Integration Key: 32 letters and digits.
- Paste it under Settings → Alerts, press Save key, then Send a test. The test opens an incident on the service and resolves it at once, so it shows on the service's timeline and pages nobody for long.
Each incident's deduplication key names the organization, the surface and the condition. If a surface fails again while its incident is open, the alert adds to that incident instead of opening a second one, and the recovery alert resolves the right one. The severity is error for a failure and warning for something late or degraded. Your service's urgency rules decide who is paged.
What fires
Each condition sends one alert when it starts and one when it clears, so a surface that stays broken does not flood the channel with an alert per check. These alerts clear each other:
backup_failed, cleared bybackup_recoveredsilentandoverdue, cleared byrecoveredunreachable, cleared byreachable_againcopy_behind, cleared bycopy_caught_up
The clearing alert carries clears, naming the condition it ends, and resolves its
PagerDuty incident.
drill_failed, drill_blocked, anomaly,
not_started, clock_skew, duplicate_host and
hosted_storage_full have no clearing alert. Resolve those incidents yourself once the
cause is fixed. Do the same for an incident on a surface you remove, or add again under a new
host. Nothing resolves it, because the clearing alert is keyed to the surface as it was.
| Event | When |
|---|---|
| backup_failed | A surface starts failing to back up. You get one alert when it starts, and another only if it fails with a different error. The daemon tries 3 times, 5 minutes apart, then waits for the next scheduled backup. |
| backup_recovered | That surface backs up again. |
| drill_failed | A Fire Drill ran and the snapshot did not restore whole. Also sent when
the drill stopped before restoring, because the host could not open storage,
drill.sandbox_url was refused, or the daemon stopped during the drill. |
| drill_blocked | A Fire Drill is due and the host cannot run it, because it does not have the private key or lacks the disk space to restore the snapshot. Sent to the webhook and shown in the console, but not emailed. |
| silent | A daemon has missed three check-ins, and at least 15 minutes have passed. Nothing is backing up its surfaces. |
| not_started | A host enrolled more than a day ago has backed up nothing since. No surface has registered from it and no daemon has reported from it. The alert says when it enrolled and what to run on it. A host enrolled only to restore gets it once, and needs no further setup. No alert is sent when the host starts. Its first backup shows in the console. |
| overdue | A surface has gone twice its schedule without a backup, whether or not its host is checking in. For a surface that checks another tool’s backups (watch:), that tool has written no new backup in twice the schedule. |
| unreachable | On three ticks in a row, the daemon could not open a surface’s database, reach its sink, or move its erasure record to the sink (checks between backups). The alert gives the error and since when. The next backup fails unless the surface answers by then. |
| reachable_again | The daemon opened that surface’s database and reached its sink again. |
| copy_behind | One of a surface's other buckets has been missing backups for a day (A second bucket). The alert names the bucket, how many backups it is missing and since when. |
| copy_caught_up | That bucket holds every backup again. |
| clock_skew | A host's clock is more than 5 minutes off from SafeGrd's server. The host's clock sets snapshot dates, which backup is kept longest, and Object Lock dates in your own bucket. Sent once a day per host. |
| duplicate_host | Two machines are taking turns reporting as one host, which happens when a config is copied to a second machine. The alert names each machine by hostname. Run safegrd enroll on the copy to give it a host of its own. Sent once a day per host. |
| anomaly | Threat Shield flagged a snapshot: a drop in rows (>20%), files (25%) or messages (30%), a dropped table, an empty snapshot, or a missing extension. The alert names the last known-good snapshot to restore from (Threat Shield). |
| kept_undrilled | A project that locks only its kept copies kept a weekly or monthly copy with no passing Fire Drill, because no backup in that week or month was drilled in time. The copy is locked. Drill it to prove it restores (kept copies). |
| drill_read_unnamed | A Fire Drill read objects the backup did not record as part of its snapshot. The drill passed, but the snapshot is not kept as a weekly or monthly copy until a drill reads only recorded objects. Upgrade safegrd on the host that took it. |
| hosted_storage_80 | Hosted storage is 80% full of locked backups. On a paid plan backups continue past 100% and the excess is billed per GB-month. On Free or a trial, new backups are refused at 100%. |
| hosted_storage_over | A paid plan’s hosted storage passed its included storage. Backups continue, and the average held above it each billing period is billed per GB-month (pricing). |
| hosted_storage_full | Hosted storage refused a backup. This happens on Free or a trial at its included storage, on a lapsed plan past its margin, or when an organization reaches its hosted storage limit. New backups are refused until locks expire or the organization moves to a larger plan or its own bucket. Existing backups stay restorable. |
| recovered | A silent host is checking in again, or an overdue surface is backing up again. |
| test | You pressed Send a test. |
Surfaces in an archived project do not alert.
What a webhook receives
Any URL other than Slack, Discord or Teams receives one JSON object per alert, sent with
Content-Type: application/json and User-Agent: SafeGrd-Alerts/1:
severity is error, warning or info. An alert that
ends a condition carries "clears" with that condition's name. For example, a
recovered after an overdue has "clears": "overdue".
Slack receives {"text": "*title*\ntext"}, Discord
{"content": "**title**\ntext"} and Teams a message with one Adaptive Card holding the
title and the text. Alerts contain operational metadata and never
include backup contents.
If a send fails, SafeGrd tries again after 1, 5 and 15 minutes, then after 1, 3 and 6 hours. It does
not retry a 4xx answer other than 408 or 429. An alert can arrive more than once, for example when
SafeGrd restarts during a send. id is the same on every attempt, so a receiver that must
act only once can skip an id it has already seen.