On one of our servers the databases are backed up every hour. The backups are encrypted with restic, an open-source backup tool, and shipped to Backblaze B2, a cloud storage service. On paper, a solid setup.
In June 2026 it stopped working. We noticed in September.
What broke
Every night the command restic forget --prune cleared out old backups. It locks the repository while it works, and that clean-up was holding the lock for hours.
The next hourly backup found the repository locked and failed. Then the one after, and the one after that. Over three months there were 7,162 failed attempts.
The worst part was that the log said "OK" regardless. Anyone opening it saw a tidy column of confirmations and no hint of trouble.
How we rebuilt it
- A new script with two modes. The hourly mode backs up the databases only. The daily mode backs up the site files and handles the clean-up of old backups, kept apart from the hourly run.
- A check on the age of the latest backup. If the newest copy is too old, a warning email goes out, whatever the log says.
- Rotated logs, archived and restarted at regular intervals, so they never grow forever and stay readable.
The check that matters
The heart of the job is the second point. Rather than asking the script whether everything went well, we ask the repository when the last copy was made. That answer does not depend on a badly written message.
A check like this takes a few lines of bash. Here is a simplified version, which assumes the restic variables (repository and password) are already set:
#!/bin/bash
# Warn if the latest backup is older than MAX_HOURS hours
MAX_HOURS=2
latest=$(restic snapshots --no-lock --latest 1 --json \
| jq -r 'sort_by(.time) | last | .time // empty')
if [ -z "$latest" ]; then
echo "No backup found" | mail -s "Backup: ERROR" [email protected]
exit 1
fi
age=$(( ( $(date +%s) - $(date -d "$latest" +%s) ) / 3600 ))
if [ "$age" -ge "$MAX_HOURS" ]; then
echo "Latest backup is $age hours old ($latest)" \
| mail -s "Backup overdue" [email protected]
fiThe script reads the date of the newest copy, compares it with the current time and only speaks up if something is wrong. If the repository does not answer at all, it raises the alarm anyway.
What if someone gets into the server?
Backups also have to survive a different risk: an attacker who gets into the server may try to delete them.
On another of our servers we closed that gap like this. The keys the server uses to write to the repository are write-only, and the repository has a 30-day lock, known as Object Lock. For a month nobody can delete or alter a copy, not even with the server's own keys.
The lesson
A backup nobody checks is just an opinion.
An "OK" message tells you, at most, that a script reached its last line. The age of the latest copy tells you whether the backup actually exists, and that is what we check.
If you have backups, ask whoever manages them two things: when was the last copy made, and who notices if they stop? If the answer to the second is "nobody", you know where to start.
