Guides / Check Coolify's backups

Know when Coolify's database backups stop working

One Coolify user found their uploads to Cloudflare R2 had stopped “about two weeks ago or so (yeah, it took me this long to notice)”. Another sees about 4% of hourly uploads to Backblaze B2 fail while the job reports “succeeded with a warning” and Validate connection stays green. A scheduled backup can stop, or upload something that does not restore, and look fine in the dashboard. This guide checks the backups Coolify already makes, first with a script, then with SafeGrd.

1. What to check

Check	Catches
The newest backup is younger than its schedule allows	Uploads that stopped: a changed key, a full bucket, a server that was rebuilt without its schedule.
It is about as large as the last one	A dump cut short in the upload, or a database that lost most of its rows before the backup.
It restores, and its tables hold rows	A file that is not a dump at all, or one `pg_restore` refuses.

Coolify writes PostgreSQL backups with pg_dump --format=custom, named pg-dump-DATABASE-UNIXTIME-ID.dmp, under data/coolify/backups/databases/TEAM/DIRECTORY/ in the bucket, and the same path on the server when S3 is off.

2. With a script

#!/usr/bin/env bash

set -euo pipefail

PREFIX=s3://my-backups/data/coolify/backups/databases/team-0/app-db/

newest=$(aws s3 ls "$PREFIX" --endpoint-url "$S3_ENDPOINT" | grep 'pg-dump-' | sort | tail -1)

name=$(echo "$newest" | awk '{print $4}')

age=$(( $(date +%s) - $(date -d "$(echo "$newest" | awk '{print $1" "$2}')" +%s) ))

[ "$age" -lt 93600 ] || { echo "newest backup is $((age / 3600)) hours old" >&2; exit 1; }

aws s3 cp "$PREFIX$name" /tmp/check.dmp --endpoint-url "$S3_ENDPOINT"

createdb restore_check

trap 'dropdb restore_check; rm -f /tmp/check.dmp' EXIT

pg_restore --single-transaction --exit-on-error --no-owner --no-acl -d restore_check /tmp/check.dmp

psql -d restore_check -Atc "SELECT count(*) FROM users"

curl -fsS https://hc-ping.com/YOUR-CHECK-ID

Run it from cron on a machine other than the Coolify server, once a day for a daily backup. The age limit is 26 hours. The last line pings a Healthchecks.io check, which alerts you when the script fails or stops running. date -d is GNU's; on macOS use gdate. Compare the size by hand, or keep yesterday's from aws s3 ls in a file. Run pg_restore from the same PostgreSQL major as the database, or a newer one.

3. With SafeGrd

Add the location as a surface on any host with the SafeGrd daemon, with a key that can only list and read the prefix:

surfaces:

- id: app-db

type: postgres

schedule: daily

watch:

url: s3://my-backups/data/coolify/backups/databases/team-0/app-db

endpoint: https://ACCOUNT.r2.cloudflarestorage.com

access_key: env:R2_KEY_ID

secret_key: env:R2_SECRET

Every day the daemon reads the newest backup once and counts each table's rows. When Coolify writes nothing for two days, the surface is Overdue and you get an alert.
Threat Shield compares each backup's rows and size with the last, and warns on a drop. A dump that does not open, or ends early, is a failed backup.
Fire Drills load the backup into a throwaway database and compare every table with the count. The certificate says Coolify made the backup.
The key stays on your host, and nothing Coolify wrote passes through SafeGrd's server.

This proves the backup restores. It does not prove the dump holds every row, because SafeGrd never reads the database itself. For that, let SafeGrd take its own backups too (Coolify backup). The details are on the PostgreSQL page.