Automate and alert on backups

A backup that depends on memory will eventually stop. You’ll be busy, then on vacation, then it’s been four months. Automation fixes that — and creates a new failure mode: jobs that break silently and keep “running” as no-ops for months. Automation must also make silence and repeated failure visible.

Wrap the backup with an honest exit

Create a wrapper result suitable for a scheduler:

if restic backup /srv/data; then
  printf '%s backup=success\n' "$(date -u +%FT%TZ)"
else
  status=$?
  printf '%s backup=failure status=%s\n' "$(date -u +%FT%TZ)" "$status" >&2
  exit "$status"
fi

The script logs a timestamped result line and, crucially, exits non-zero on failure. Schedulers and alerting can only react to what the script reports.

Schedule it with cron, capturing output:

# /etc/cron.d/backup
0 3 * * * root /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Cron is where silent failures breed. It runs with a minimal environment: a stripped PATH, no RESTIC_REPOSITORY, no RESTIC_PASSWORD_FILE. A script that works perfectly in your shell fails at 3 AM because the variables were only in your session. Export everything the job needs inside the script itself, and test by running it from a bare environment: env -i /usr/local/bin/backup.sh.

Test the failure path

Now force a permission and destination failure — make the repository unreachable, or revoke read access to /srv/data. Confirm the scheduler records non-zero status and the alert reaches the responsible person. An alert channel nobody checks is decoration. If you’ve never seen your backup alert fire, you don’t have alerting; you have optimism.

Alert on silence, not just errors

There’s a failure the error path can’t catch: the job never ran at all. Disabled cron, powered-off machine, deleted crontab. No run means no error means no alert.

A “job did not run” condition needs detection too. Alert on missing recent successful snapshots. The dead-man’s-switch pattern works well: the wrapper pings a monitoring URL after each success, and the monitor alerts when pings stop arriving:

restic backup /srv/data && curl -fsS https://hc-ping.com/6c2ea1d4-backup

Alternatively, a daily check that the newest snapshot is younger than 24 hours (restic snapshots --json plus a timestamp comparison) catches the same silence from the repository side.