Skip to main content

ClickHouse cluster fault tolerance

ClickHouse cluster fault tolerance is affected by:

Data replication

If a shard consists of only one node, data in it is not replicated. To ensure data replication, add additional nodes—replicas—to the shard.

You can specify the number of nodes in a shard when creating a cluster. You can change the number of nodes in a shard in an existing cluster — add or delete. The total number of nodes in a shard cannot exceed ten.

Node allocation type

The node allocation type depends on the number of nodes in shards, the type of pool where the cluster is located, and the number of segments in the pool:

  • Single-Segment — in one segment of a single-zone pool. Applicable:

    • clusters with one shard containing only one node;
    • shards with multiple nodes, if the cluster is located in a single-segment pool;
    • shards with multiple nodes in HighFreq configurations;
note

For example, you created a cluster in pool ru-2 with ClickHouse Keeper and a shard containing two nodes. For both node groups, you used configurations from the HighFreq line. All nodes will be placed in one segment of the ru-2c pool.

  • Multi-Segment — in different segments of a single-zone pool. Applicable to clusters that consist of shards with multiple nodes if the cluster is located in a pool with multiple segments. Segments are located in different racks of the same data center. Nodes of each shard are distributed across segments sequentially;
note

For example, you created a cluster in pool ru-1 with one shard, in which there are four nodes. The first three nodes will be placed sequentially in segments of pools ru-1a, ru-1b, and ru-1c. The fourth node will be placed in segment ru-1a. If you add a fifth node, it will be placed in segment ru-1b.

  • Single-AZ — in one segment of a multi-zone pool. Applicable to clusters that consist of one shard with one node if the cluster is located in pool ru-6. If you increase the number of nodes in the shard, the node placement type will change to Multi-AZ;

  • Multi-AZ — in different segments of a multi-zone pool. Applicable to clusters that consist of shards with multiple nodes if the cluster is located in pool ru-6. Each pool segment is located in a separate data center. Nodes of each shard are distributed across segments sequentially.

note

For example, you created a cluster in pool ru-6 with one shard, in which there are four nodes. The first three nodes will be placed sequentially in segments of pools ru-6a, ru-6b, and ru-6c. The fourth node will be placed in segment ru-6a. If you add a fifth node, it will be placed in segment ru-6b.

You can view the list of pools and segments in the Servercore Infrastructure table in the Locations: countries, regions, availability zones, pools, and data center addresses guide.

Add a node

A shard can contain no more than ten nodes. During the node addition process, other nodes in the shard will continue to operate for read and write operations.

  1. If the cluster consists of only one shard with one node, add ClickHouse Keeper.
  2. In the Dashboard, on the top menu, click Products and select Managed Databases.
  3. Open the Active tab.
  4. Open the cluster page → Settings tab.
  5. In the shard menu, select Change node count.
  6. Open the Add tab.
  7. Specify the new number of nodes in the shard. If there are no free addresses in the subnet to which the cluster is connected, you cannot add a node, as each new node occupies a new address in the subnet.
  8. Click Save.

Delete a node

carefully

Deleting a node will result in the loss of all data stored on that node.

During deletion, other nodes in the shard will continue to operate in read and write modes. The public IP address assigned to the node will be deleted.

  1. In the Dashboard, on the top menu, click Products and select Managed Databases.
  2. Open the Active tab.
  3. Open the cluster page → Settings tab.
  4. In the shard menu, select Change node count.
  5. Open the Delete tab.
  6. Select the node you want to delete. There must be at least one node remaining in the shard. You cannot delete all nodes from a shard.
  7. Click Save.