ClickHouse Cluster Deployment

This article explains how to use clickhouse_ansible to perform a ClickHouse cluster or standalone mode deployment.

The commands in this article are based on the portable Ansible distribution by default and have performed the following initialization:

/usr/local/dbbot/bin/dbbotctl env setup
source ~/.bashrc

Once completed, you can use ansible-playbook and ansible directly.

1. Scope of application

Supports the following entry playbooks:

  • playbooks/deploy_cluster.yml: cluster deployment.
  • playbooks/deploy_single.yml: rapid deployment for a single node or an experimental environment.

The default documentation example uses 3 shard 2 replicas, but the inventory design supports expansion to other x shard y replica topologies.

2. Preconditions

It is recommended to complete the following preparations before deployment:

  1. Install dbbot on the control node and complete the portable distribution Ansible registration.
  2. Plan the ClickHouse node and ZooKeeper/Keeper node.
  3. Confirm that the target node network, time zone, disk directory, and operating system baseline meet the requirements.
  4. Confirm that the control node can SSH to the target host without interaction.
  5. Prepare the ClickHouse installation package, or confirm that the control node can connect to the Internet to download the required package.

clickhouse_ansible splits inventory by scenario by default:

  • inventory/hosts.deploy.ini: Deployment-specific.
  • inventory/hosts.dr_deploy.ini: dedicated for disaster recovery cluster deployment.
  • inventory/hosts.backup.ini: dedicated for backup.
  • inventory/hosts.dr_backup.ini: dedicated for disaster recovery cluster backup disk preparation.
  • inventory/hosts.restore.ini: dedicated to recovery.
  • inventory/hosts.nfs_server.ini: lab/test NFS server only; dbbot does not deploy production NFS.
  • inventory/hosts.single.ini: dedicated for single-node validation.

In deployment scenarios, it is recommended to set it in [all:vars]:

dbbot_inventory_purpose=deploy
ansible_python_interpreter=auto_silent
ansible_user=root
ansible_ssh_pass="'<your_ssh_password>'"

4. Deploy inventory minimal example

[clickhouse_deploy]
ck-src-11-1 ansible_host=192.0.2.11 shard=1 replica=1 instance_id=1 replica_name=ck-src-s1-r1 system_hostname=ck-src-node1
ck-src-11-2 ansible_host=192.0.2.11 shard=3 replica=2 instance_id=2 replica_name=ck-src-s3-r2 system_hostname=ck-src-node1
ck-src-12-1 ansible_host=192.0.2.12 shard=1 replica=2 instance_id=1 replica_name=ck-src-s1-r2 system_hostname=ck-src-node2
ck-src-12-2 ansible_host=192.0.2.12 shard=2 replica=1 instance_id=2 replica_name=ck-src-s2-r1 system_hostname=ck-src-node2
ck-src-13-1 ansible_host=192.0.2.13 shard=2 replica=2 instance_id=1 replica_name=ck-src-s2-r2 system_hostname=ck-src-node3
ck-src-13-2 ansible_host=192.0.2.13 shard=3 replica=1 instance_id=2 replica_name=ck-src-s3-r1 system_hostname=ck-src-node3

[clickhouse_cluster:children]
clickhouse_deploy

[zookeeper_deploy]
ck-src-11-1 zk_id=1
ck-src-12-1 zk_id=2
ck-src-13-1 zk_id=3

[zookeeper_cluster:children]
zookeeper_deploy

[all:vars]
dbbot_inventory_purpose=deploy
ansible_python_interpreter=auto_silent
ansible_user=root
ansible_ssh_pass="'<your_ssh_password>'"

Description:

  • instance_id is recommended to be filled in explicitly to facilitate port, directory and service name derivation.
  • replica_name is optional; if omitted, it is derived automatically from shard / replica.
  • If fcs_set_hostname=true is enabled, system_hostname can be additionally configured for the host line.
  • These three fields are deployment-oriented. Minimal backup and restore inventories do not require them; see ClickHouse Restore for the recovery-side layout.
  • In Keeper mode, it is still necessary to ensure that the coordination node is completely defined.

5. Key variable file

Before deployment, focus on the following files:

  • playbooks/vars/common_config.yml: Common parameters such as version, user, directory, password, Keeper switch, etc.
  • playbooks/vars/cluster_config.yml: cluster name, port baseline and topology related parameters.
  • playbooks/vars/standalone_config.yml: Standalone mode behavior.

Common parameters that need to be confirmed:

  • clickhouse_version
  • clickhouse_cluster_name
  • use_clickhouse_keeper
  • clickhouse_enable_ssl
  • clickhouse_install_debug_package
  • clickhouse_default_password
  • clickhouse_default_user_allowed_networks
  • clickhouse_default_user_auto_include_cluster_nodes
  • fcs_allow_dbbot_default_passwd
  • fcs_set_hostname
  • deploy_require_manual_confirm

For a more complete parameter description, see: ClickHouse variable and parameter description

Password guardrails:

  • The public default password for clickhouse_default_password is Dbbot_default@8888.
  • Default fcs_allow_dbbot_default_passwd: false, so deployment-, backup-, and restore-related playbooks will intercept this public default password in the pre_tasks stage.
  • If you want to verify quickly in the experimental environment, you can explicitly set fcs_allow_dbbot_default_passwd: true; this is not recommended in the production environment.
  • Passwords may contain XML- and shell-significant characters such as $, &, <, >, and quotes. dbbot XML-escapes password values before rendering configuration.
  • dbbot-managed clickhouse-client calls read credentials from /etc/clickhouse-server[-N]/dbbot-client.xml instead of putting the password in process arguments. This file is owned by root:root with mode 0600.
  • config.d/cluster.xml, which contains remote_servers credentials, is owned by the ClickHouse runtime account with mode 0600.

Trusted-source behavior:

  • The default user allows 127.0.0.1 and ::1 by default, then automatically appends the exact ansible_host IP of every ClickHouse node in the deployment inventory.
  • To add a management subnet, disable automatic inclusion, or explicitly allow every source, edit playbooks/vars/common_config.yml.
  • See “Trusted sources for the default user” in ClickHouse Variables and Parameters for complete examples.

6. Prepare installation package

If you turn off automatic downloading, you need to put the installation package into downloads/ in advance:

cd /usr/local/dbbot/clickhouse_ansible/downloads
wget "https://packages.clickhouse.com/tgz/stable/clickhouse-common-static-23.6.1.1524-amd64.tgz"
wget "https://packages.clickhouse.com/tgz/stable/clickhouse-server-23.6.1.1524-amd64.tgz"
wget "https://packages.clickhouse.com/tgz/stable/clickhouse-client-23.6.1.1524-amd64.tgz"
wget "https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz"

Description:

  • clickhouse-common-static-dbg has been changed to optional, requiring preparation and installation only when clickhouse_install_debug_package=true.
  • It is recommended to keep clickhouse_install_debug_package=false by default for official releases to reduce package size and distribution time.

7. Execute deployment

7.1 Grammar check

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.deploy.ini \
  deploy_cluster.yml --syntax-check

7.2 Cluster deployment

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.deploy.ini \
  deploy_cluster.yml

7.3 Keeper mode deployment

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.deploy.ini \
  deploy_cluster.yml \
  -e "use_clickhouse_keeper=true"

7.4 Stand-alone mode deployment

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.single.ini \
  deploy_single.yml

8. System changes

Deploying a playbook not only installs ClickHouse, but also performs system-side initialization on the target host. Defaults may involve:

  1. Set the time zone to Asia/Shanghai
  2. Modify the mapping between hostname and /etc/hosts
  3. Disable swap and comment the swap entry in /etc/fstab
  4. Adjust limits.conf / limits.d
  5. Adjust ClickHouse-specific parameters under sysctl.d
  6. Tweak THP, SELinux and Firewall

If you don’t want something to take effect, at least evaluate in advance:

  • fcs_set_hostname
  • clickhouse_uid / clickhouse_gid
  • fcs_auto_download_packages
  • clickhouse_install_debug_package

9. Post-deployment validation

8.1 Port monitoring

ansible \
  all -i ../inventory/hosts.deploy.ini -m shell \
  -a "ss -lntp | egrep ':8123 |:8124 |:8443 |:8444 |:9000 |:9001 |:9440 |:9441 '"

8.2 Cluster topology

SELECT cluster, shard_num, replica_num, host_name, port
FROM system.clusters
WHERE cluster = 'example_3shards_2replicas'
ORDER BY shard_num, replica_num;

8.3 Copy status

SELECT table, is_readonly, queue_size, absolute_delay
FROM system.replicas
ORDER BY table;

10. Common precautions

  1. Deploy the default requirement dbbot_inventory_purpose=deploy to avoid misuse of backup or recovery inventory.
  2. ansible_ssh_pass is recommended to be written in "'password'" to avoid Ansible inventory parsing exceptions.
  3. Manual access control is enabled by default and is not recommended to be closed for a long time during production.
  4. If you do not want to modify the system hostname, you can set fcs_set_hostname to false.
  5. If the target machine already has an account with the same UID/GID, you need to adjust clickhouse_uid / clickhouse_gid first.
  6. If you want a full drill after deployment, read ClickHouse Data Generation and Drill Tools first, then continue with ClickHouse Backup, ClickHouse Restore, and ClickHouse Data Comparison and Restore Validation.