Back up, monitor, and restore

Replicas are not backups. Replication copies your mistakes to every server within seconds. A dropped partition, a wrong ALTER TABLE ... DELETE, a truncate on the wrong environment. Only a backup holds the state from before the mistake.

Back up everything the recovery needs: tables, metadata, users, and configuration. ClickHouse has native commands for the data and the schema:

BACKUP TABLE analytics.events
TO Disk('backups', 'events-2026-08-03.zip');

The backups destination is a disk you configure in the server’s storage settings. S3-compatible object storage is the other common target. The backup contains both schema and data, and later backups to the same destination can be incremental.

Don’t forget what a table backup misses. Users, roles, and grants need their own copy, and so do the server configuration files. Otherwise you restore a cluster full of data that nobody can log in to query.

Keep the raw event source around too, when that’s part of the design. A queue or object store that retains 30 days of events is a recovery path for recent data all by itself.

Monitor the things that fail quietly

Watch query failures, ingestion lag, disk, parts, merges, replication queues, memory, and backup jobs. Most of this comes straight from system tables. Failed queries live in system.query_log, part counts in system.parts, replication debt in system.replication_queue. Every backup’s outcome is in system.backups:

SELECT name, status, error
FROM system.backups
ORDER BY start_time DESC
LIMIT 5;

A backup job that has been failing for three weeks is the same as no backup at all, and you always find out at the worst moment. Alert on status != 'BACKUP_CREATED', not just on the job having run.

A restore you haven’t run is a hypothesis

Restore into an isolated database or cluster. Never onto the production table first:

RESTORE TABLE analytics.events AS analytics.events_restored
FROM Disk('backups', 'events-2026-08-03.zip');

Then compare row counts, one aggregate, permissions, and a real dashboard result before you call the recovery good:

SELECT count() FROM analytics.events_restored;
SELECT service, count() FROM analytics.events_restored
GROUP BY service ORDER BY service;

The count tells you the data arrived. The aggregate tells you it’s the right data. Logging in with the dashboard user tells you access survived. Loading one real dashboard against the restored copy tells you the whole chain works.

Run this drill on a schedule, not during an incident. The point of rehearsing is finding the missing grant or the misconfigured backup disk on a calm Tuesday, when it’s a ticket and not an outage. Put it in the calendar once a month. My rule is that a backup I haven’t restored doesn’t count.