Architecture

ClickHouse Keeper: Setup, Monitoring and Migrating from ZooKeeper

Every replicated ClickHouse table depends on a coordination service. When it is slow, inserts slow down; when it loses quorum, tables go read-only. ClickHouse Keeper is the native replacement for ZooKeeper. Here is how to run it, what to watch, and how to migrate without losing metadata.

· 12 min read · AG Data Team

ClickHouse Keeper is the coordination service that ReplicatedMergeTree tables and ON CLUSTER DDL depend on. It speaks the ZooKeeper client protocol, uses Raft for consensus, and ships in the same package as ClickHouse. If you run replicated ClickHouse, Keeper (or ZooKeeper) is the most important component you probably do not monitor.

What Keeper does and how it differs from ZooKeeper

Replicas register parts, merges and mutations in Keeper, and ON CLUSTER queries are queued there. Keeper is written in C++, uses the NuRaft implementation of Raft, and by default gives linearizable writes and non-linearizable reads, the same guarantees as ZooKeeper. Standard ZooKeeper clients can connect to it (ClickHouse Keeper docs).

The differences that matter in practice:

Embedded or standalone?

Keeper can run inside clickhouse-server (add a <keeper_server> section to the server config) or as a separate clickhouse-keeper process. Embedded is fine for small clusters and test setups. For production we run Keeper standalone, on its own three small hosts. That is our experience rather than a documented rule, but the reasons are concrete: a heavy query or OOM on a ClickHouse node then cannot take a Keeper member down with it, and you can restart ClickHouse without touching quorum. The docs themselves advise storing coordination logs on non-busy nodes.

A three-node configuration

<clickhouse>
  <keeper_server>
    <tcp_port>9181</tcp_port>          <!-- default is 2181 -->
    <server_id>1</server_id>           <!-- unique per node: 1, 2, 3 -->
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
    <coordination_settings>
      <operation_timeout_ms>10000</operation_timeout_ms>
      <session_timeout_ms>30000</session_timeout_ms>
    </coordination_settings>
    <raft_configuration>
      <server><id>1</id><hostname>keeper1.internal</hostname><port>9234</port></server>
      <server><id>2</id><hostname>keeper2.internal</hostname><port>9234</port></server>
      <server><id>3</id><hostname>keeper3.internal</hostname><port>9234</port></server>
    </raft_configuration>
  </keeper_server>
</clickhouse>

Then point every ClickHouse server at all Keeper nodes:

<!-- on every ClickHouse server -->
<zookeeper>
  <node><host>keeper1.internal</host><port>9181</port></node>
  <node><host>keeper2.internal</host><port>9181</port></node>
  <node><host>keeper3.internal</host><port>9181</port></node>
</zookeeper>

Quorum: why three, and what one node means

Raft needs a majority of members to accept writes. A three-node Keeper cluster keeps working when one node fails; a five-node cluster survives two. Two nodes are worse than one: losing either stops writes. A single node is acceptable only where losing coordination for a while is acceptable.

No quorum means no replicated writes. Without a working Keeper majority, an INSERT into a replicated table throws an exception, and a server that starts while Keeper is unavailable switches its replicated tables to read-only mode (replication docs). We cover that failure from the ClickHouse side in "Table is in readonly mode".

Monitoring Keeper

# is it alive, and what role does it have?
echo ruok | nc keeper1.internal 9181      # imok
echo isro | nc keeper1.internal 9181      # rw / ro
echo mntr | nc keeper1.internal 9181 | grep -E \
  'zk_server_state|zk_followers|zk_synced_followers|zk_avg_latency|zk_max_latency|zk_outstanding_requests|zk_znode_count|zk_approximate_data_size'

-- from ClickHouse: which Keeper node each server is connected to
SELECT name, host, port, connected_time, is_expired, session_timeout_ms
FROM system.zookeeper_connection;

What we alert on: no leader or more than one; zk_synced_followers below the expected count; zk_outstanding_requests staying above zero; zk_max_latency rising over days; free disk on the log and snapshot paths.

Migrating from ZooKeeper

There is no rolling migration. The documented procedure needs a short write freeze:

  1. Stop ingestion on all ClickHouse nodes and stop background work such as merges.
  2. Stop every ZooKeeper node. Optionally start and stop the leader once to force a consistent snapshot.
  3. Convert the data with clickhouse-keeper-converter (needs ZooKeeper 3.4 or later).
  4. Copy the resulting snapshot to all Keeper nodes before any of them starts.
  5. Point ClickHouse at Keeper, start Keeper on all nodes, restart ClickHouse.
  6. Compare metrics with your pre-migration baseline, then resume merges and ingestion.
# all ClickHouse nodes: stop inserts, then
SYSTEM STOP MERGES;
# stop every ZooKeeper node, then convert
clickhouse-keeper-converter \
  --zookeeper-logs-dir /var/lib/zookeeper/version-2 \
  --zookeeper-snapshots-dir /var/lib/zookeeper/version-2 \
  --output-dir /var/lib/clickhouse/coordination/snapshots
# copy the snapshot to ALL Keeper nodes before starting any of them

Rehearse it on a copy first. The conversion itself is quick; the downtime is mostly the coordinated stop and restart of every node.

Recovering after losing quorum

If two of three nodes are gone for good, Raft cannot reconfigure itself. Keeper has a recovery mode for this, and the docs call it a last resort: make sure the failed nodes cannot come back, pick the node with the most up-to-date state, back up its log and snapshot directories, update the configuration on all intended members, then enable recovery on that node with rcvr or a restart with --force-recovery. Start the new nodes one at a time, waiting until mntr reports follower before starting the next.

Two more warnings from the docs: changing a node's hostname is the same as removing and re-adding the server, which may be impossible without quorum, and you should back up logs and snapshots before changing the disk layout.

On large clusters: where Keeper hurts

A three-node test cluster with a few tables never stresses Keeper. These are the problems we see at scale:

Key takeaways

ClickHouse ClickHouse Keeper ZooKeeper Replication
Work with us

Keeper or ZooKeeper causing replication trouble?

We design and operate ClickHouse Keeper for production clusters, migrate from ZooKeeper, and recover clusters that lost quorum or went read-only.

Get in touch More from the blog