ClickHouse Keeper is the coordination service that ReplicatedMergeTree tables and ON CLUSTER DDL depend on. It speaks the ZooKeeper client protocol, uses Raft for consensus, and ships in the same package as ClickHouse. If you run replicated ClickHouse, Keeper (or ZooKeeper) is the most important component you probably do not monitor.
What Keeper does and how it differs from ZooKeeper
Replicas register parts, merges and mutations in Keeper, and ON CLUSTER queries are queued there. Keeper is written in C++, uses the NuRaft implementation of Raft, and by default gives linearizable writes and non-linearizable reads, the same guarantees as ZooKeeper. Standard ZooKeeper clients can connect to it (ClickHouse Keeper docs).
The differences that matter in practice:
- Snapshot and log formats are incompatible with ZooKeeper, and so is the interserver protocol. A mixed ZooKeeper and Keeper ensemble is impossible.
- Some ZooKeeper features are missing:
CONTAINERznodes, children under TTL nodes, SASL authentication. ACL schemesworld,authanddigestwork. - Keeper is not something you configure in ClickHouse Cloud; this article is about self-managed clusters.
Embedded or standalone?
Keeper can run inside clickhouse-server (add a <keeper_server> section to the server config) or as a separate clickhouse-keeper process. Embedded is fine for small clusters and test setups. For production we run Keeper standalone, on its own three small hosts. That is our experience rather than a documented rule, but the reasons are concrete: a heavy query or OOM on a ClickHouse node then cannot take a Keeper member down with it, and you can restart ClickHouse without touching quorum. The docs themselves advise storing coordination logs on non-busy nodes.
A three-node configuration
<clickhouse>
<keeper_server>
<tcp_port>9181</tcp_port> <!-- default is 2181 -->
<server_id>1</server_id> <!-- unique per node: 1, 2, 3 -->
<log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
<snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
<coordination_settings>
<operation_timeout_ms>10000</operation_timeout_ms>
<session_timeout_ms>30000</session_timeout_ms>
</coordination_settings>
<raft_configuration>
<server><id>1</id><hostname>keeper1.internal</hostname><port>9234</port></server>
<server><id>2</id><hostname>keeper2.internal</hostname><port>9234</port></server>
<server><id>3</id><hostname>keeper3.internal</hostname><port>9234</port></server>
</raft_configuration>
</keeper_server>
</clickhouse>
server_idmust be unique and must match theidof that node inraft_configuration. Keep the id-to-hostname mapping stable, and use hostnames, not IPs.- The default client port is 2181, the same as ZooKeeper; the ClickHouse examples use 9181. Port 9234 in the example is the Raft port between Keeper nodes.
operation_timeout_msdefaults to 10000.session_timeout_msis the maximum client session timeout (default 100000, minimum 10000).- Other defaults worth knowing:
snapshot_distanceandreserved_log_itemsare both 100000,snapshots_to_keepis 3, anddigest_enabled(a real-time consistency check) is on.
Then point every ClickHouse server at all Keeper nodes:
<!-- on every ClickHouse server -->
<zookeeper>
<node><host>keeper1.internal</host><port>9181</port></node>
<node><host>keeper2.internal</host><port>9181</port></node>
<node><host>keeper3.internal</host><port>9181</port></node>
</zookeeper>
Quorum: why three, and what one node means
Raft needs a majority of members to accept writes. A three-node Keeper cluster keeps working when one node fails; a five-node cluster survives two. Two nodes are worse than one: losing either stops writes. A single node is acceptable only where losing coordination for a while is acceptable.
Monitoring Keeper
# is it alive, and what role does it have?
echo ruok | nc keeper1.internal 9181 # imok
echo isro | nc keeper1.internal 9181 # rw / ro
echo mntr | nc keeper1.internal 9181 | grep -E \
'zk_server_state|zk_followers|zk_synced_followers|zk_avg_latency|zk_max_latency|zk_outstanding_requests|zk_znode_count|zk_approximate_data_size'
-- from ClickHouse: which Keeper node each server is connected to
SELECT name, host, port, connected_time, is_expired, session_timeout_ms
FROM system.zookeeper_connection;
ruokreturnsimokif the server is running;isroreturnsroorrw;mntrreturns metrics includingzk_server_state(leader, follower),zk_followersandzk_synced_followers, latency, outstanding requests, znode count and approximate data size.- Four-letter-word commands must be in
four_letter_word_white_list. The list is read at startup, so changing it needs a restart. The default list already includesruok,mntr,isro,statandsrvr. - For Kubernetes probes,
http_controlexposes an HTTP readiness endpoint such as/ready. - On the ClickHouse side,
system.zookeeper_connectionshows which Keeper node each server is connected to, whether the session expired and the session timeout (system table docs).
What we alert on: no leader or more than one; zk_synced_followers below the expected count; zk_outstanding_requests staying above zero; zk_max_latency rising over days; free disk on the log and snapshot paths.
Migrating from ZooKeeper
There is no rolling migration. The documented procedure needs a short write freeze:
- Stop ingestion on all ClickHouse nodes and stop background work such as merges.
- Stop every ZooKeeper node. Optionally start and stop the leader once to force a consistent snapshot.
- Convert the data with
clickhouse-keeper-converter(needs ZooKeeper 3.4 or later). - Copy the resulting snapshot to all Keeper nodes before any of them starts.
- Point ClickHouse at Keeper, start Keeper on all nodes, restart ClickHouse.
- Compare metrics with your pre-migration baseline, then resume merges and ingestion.
# all ClickHouse nodes: stop inserts, then
SYSTEM STOP MERGES;
# stop every ZooKeeper node, then convert
clickhouse-keeper-converter \
--zookeeper-logs-dir /var/lib/zookeeper/version-2 \
--zookeeper-snapshots-dir /var/lib/zookeeper/version-2 \
--output-dir /var/lib/clickhouse/coordination/snapshots
# copy the snapshot to ALL Keeper nodes before starting any of them
Rehearse it on a copy first. The conversion itself is quick; the downtime is mostly the coordinated stop and restart of every node.
Recovering after losing quorum
If two of three nodes are gone for good, Raft cannot reconfigure itself. Keeper has a recovery mode for this, and the docs call it a last resort: make sure the failed nodes cannot come back, pick the node with the most up-to-date state, back up its log and snapshot directories, update the configuration on all intended members, then enable recovery on that node with rcvr or a restart with --force-recovery. Start the new nodes one at a time, waiting until mntr reports follower before starting the next.
Two more warnings from the docs: changing a node's hostname is the same as removing and re-adding the server, which may be impossible without quorum, and you should back up logs and snapshots before changing the disk layout.
On large clusters: where Keeper hurts
A three-node test cluster with a few tables never stresses Keeper. These are the problems we see at scale:
- Load follows parts. Each INSERT into a replicated table adds approximately ten entries to Keeper, and the replication docs recommend batching inserts, no more than one INSERT per second. Merges and fetches are coordinated there too. Many small inserts across hundreds of tables means a steady stream of Keeper writes. Batching inserts helps ClickHouse and Keeper at once, see "Too many parts".
- Disk latency is write latency. With
force_syncon (the default), the coordination log must be on a local disk, and every write waits for it. A noisy or IOPS-capped volume shows up directly as slower inserts and higherzk_max_latency. - Memory and snapshots grow with znodes. In our experience Keeper memory tracks
zk_znode_countandzk_approximate_data_size, and snapshots grow with them. Leftover metadata from dropped tables and dead replicas adds up; clean up replicas you have removed. - ON CLUSTER queues. Distributed DDL runs through Keeper. A stuck node can leave tasks waiting, and on big clusters these queues are worth monitoring separately.
- Restarts take longer. Bigger logs and snapshots mean longer startup; the docs suggest lowering
log_startup_read_max_streamson seek-bound storage. Plan rolling restarts one node at a time, and confirm the node is back as follower before touching the next.
Key takeaways
- ClickHouse Keeper is a Raft-based, ZooKeeper-compatible coordination service; it cannot share an ensemble with ZooKeeper.
- Run three (or five) nodes, ideally standalone on their own hosts, with stable ids and hostnames.
- Monitor with
ruok,isro,mntrandsystem.zookeeper_connection; alert on leader state, synced followers, latency and outstanding requests. - Migration from ZooKeeper needs a write freeze and
clickhouse-keeper-converter; copy the snapshot to every node before starting any. --force-recoveryis a last resort after permanent quorum loss; back up the surviving node first.- At scale, Keeper load follows insert and part volume, and its latency follows disk latency.