ClickHouse集群+ZooKeeper集群架构下的ZooKeeper集群迁移
目前测试环境的ck集群+zk集群架构如下:
ClickHouse | ZooKeeper |
192.168.1.171 192.168.1.172 192.168.1.173 192.168.1.174 | 192.168.1.171 192.168.1.172 192.168.1.173 |
由于ck和zk混搭在一块,存在较为严重的性能问题,计划将zk从171,172,173迁移到34,35,36上。
通过向现有的zk集群中添加成员34,35,36变为6个节点的集群后,在剔除171,172,173这三个成员达到迁移目的,即扩容然后缩容。
最终结构如下:
ClickHouse | ZooKeeper |
192.168.1.171 192.168.1.172 192.168.1.173 192.168.1.174 | 192.168.1.34 192.168.1.35 192.168.1.36
|
启用动态配置功能
由于目前zk(171,172,173)并没有启动动态配置功能,需要停机修改配置。
先停止ck全部节点,在停止zk全部节点。(略)
修改171,172,173的zk配置:
vi /usr/local/zookeeper/conf/zoo.cfg
删除server列表配置:
server.1=192.168.1.171:2888:3888
server.2=192.168.1.172:2888:3888
server.3=192.168.1.173:2888:3888
追加:
skipACL=yes
reconfigEnabled=true
dynamicConfigFile=/usr/local/zookeeper/conf/zoo.cfg.dynamic
添加文件zoo.cfg.dynamic,指定原来的server列表节点:
vi /usr/local/zookeeper/conf/zoo.cfg.dynamic
server.1=192.168.1.171:2888:3888:participant;2181
server.2=192.168.1.172:2888:3888:participant;2181
server.3=192.168.1.173:2888:3888:participant;2181
启动171,172,173的zk,然后启动ck集群。(略)
安装配置zk服务器
按照企业规范对新的34,35,36安装配置zk软件,但先不启动zk服务(略)。
特别需要注意:ck集群(171-174),旧zk集群(171-173),新zk集群(34-36)的/etc/hosts要对应有全部节点的解析。
cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.171 dev-app76
192.168.1.172 dev-app77
192.168.1.173 dev-app78
192.168.1.174 dev-app79
192.168.1.34 zk01
192.168.1.35 zk02
192.168.1.36 zk03
zk集群扩容
添加新节点zk01(34)
vi /usr/local/zookeeper/conf/zoo.cfg.dynamic
server.1=192.168.1.171:2888:3888:participant;2181
server.2=192.168.1.172:2888:3888:participant;2181
server.3=192.168.1.173:2888:3888:participant;2181
server.4=192.168.1.34:2888:3888:participant;2181
启动zk01:
zkServer.sh start
或者:systemctl start zookeeper
旧的zk服务器171,172,173随便挑选一台执行。
[zk: localhost:2181(CONNECTED) 0] reconfig -add 4=192.168.1.34:2888:3888;2181
Committed new configuration:
server.1=192.168.1.171:2888:3888:participant;0.0.0.0:2181
server.2=192.168.1.172:2888:3888:participant;0.0.0.0:2181
server.3=192.168.1.173:2888:3888:participant;0.0.0.0:2181
server.4=192.168.1.34:2888:3888:participant;0.0.0.0:2181
version=630000e4be
检查各个节点zk状态。
echo conf | nc zk01 2181 | grep server.
echo conf | nc 192.168.1.171 2181 | grep server.
echo conf | nc 192.168.1.172 2181 | grep server.
echo conf | nc 192.168.1.173 2181 | grep server.
样例:
[root@zk01 conf]# echo conf | nc zk01 2181 | grep server.
serverId=4
server.4=192.168.1.34:2888:3888:participant;0.0.0.0:2181
server.5=192.168.1.35:2888:3888:participant;0.0.0.0:2181
server.6=192.168.1.36:2888:3888:participant;0.0.0.0:2181
View Code
检查ck读写是否异常。
参考语句:(关于gv$parts,参考clickhouse集群跨节点查询其他节点系统表信息)
CREATE TABLE default.zkm on cluster ceb_cluster
(
`id` String DEFAULT 'NULL' COMMENT '用户编号',
`repo` String DEFAULT 'NULL' COMMENT '仓库编号'
)
ENGINE = Distributed('ceb_cluster', 'default', 'zkm_local', rand());
CREATE TABLE default.zkm_local on cluster ceb_cluster
(
`id` String DEFAULT 'NULL' COMMENT '用户编号',
`repo` String DEFAULT 'NULL' COMMENT '仓库编号'
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/ceb_cluster-{shard}/default/zkm_local', '{replica}')
ORDER BY id
SETTINGS index_granularity = 8192;
insert into zkm values('a','b');
insert into zkm select * from zkm settings insert_deduplicate=0; --这条多次执行
SELECT
host AS `主机名`,
sum(rows) AS `总行数`,
formatReadableSize(sum(data_uncompressed_bytes)) AS `原始大小`,
formatReadableSize(sum(data_compressed_bytes)) AS `压缩大小`,
round((sum(data_compressed_bytes) / sum(data_uncompressed_bytes)) * 100, 0) AS `压缩率`
FROM `gv$parts`
WHERE (database = 'default') AND (table = 'zkm_local') AND (active = 1) AND (engine = 'ReplicatedMergeTree')
GROUP BY host
ORDER BY host ASC;
View Code
添加新节点zk02(35)
同上诉添加zk01。略。
添加新节点zk03(36)
同上诉添加zk01。略。
修改ck服务器zk配置
将ck作为客户端连接的原zk(171,172,173)替换为新的zk节点(34,35,36)。
所有的ck(171,172,173,174)节点:
vi /etc/clickhouse-server/config.d/zks.xml
<?xml version="1.0"?>
<clickhouse>
<zookeeper>
<node index="1">
<host>zk01</host>
<port>2181</port>
</node>
<node index="2">
<host>zk02</host>
<port>2181</port>
</node>
<node index="3">
<host>zk03</host>
<port>2181</port>
</node>
</zookeeper>
</clickhouse>
服务器跟踪配置文件中的更改,以及执行替换和复盖时使用的文件和ZooKeeper节点,并动态重新加载用户和集群的设置。 这意味着您可以在不重新启动服务器的情况下修改群集、用户及其设置。
确认新配置信息是否实时生效(在ck的数据文件所在根目录):
cat /clickhouse/preprocessed_configs/config.xml
zk集群缩容
从zk服务器34,35,36随便挑选一台执行。
zkCli.sh进去交互式:
reconfig -remove 1
reconfig -remove 2
reconfig -remove 3
检查各个节点zk状态。
echo conf | nc zk01 2181 | grep server.
echo conf | nc zk02 2181 | grep server.
echo conf | nc zk03 2181 | grep server.
echo conf | nc 192.168.1.171 2181 | grep server.
echo conf | nc 192.168.1.172 2181 | grep server.
echo conf | nc 192.168.1.173 2181 | grep server.
关闭旧的zk节点(171,172,173):
zkServer.sh stop
检查各个节点zk状态。
echo conf | nc zk01 2181 | grep server.
echo conf | nc zk02 2181 | grep server.
echo conf | nc zk03 2181 | grep server.
检查ck是否有会话连接到zk:
echo cons | nc zk01 2181
echo cons | nc zk02 2181
echo cons | nc zk03 2181
或者
echo srvr | nc zk01 2181
echo srvr | nc zk02 2181
echo srvr | nc zk03 2181
最终如下:
[root@zk03 clickhouse]# echo conf | nc zk01 2181 | grep server.
serverId=4
server.4=192.168.1.34:2888:3888:participant;0.0.0.0:2181
server.5=192.168.1.35:2888:3888:participant;0.0.0.0:2181
server.6=192.168.1.36:2888:3888:participant;0.0.0.0:2181
卸载旧的集群zk节点171,172,173:
绿色安装的zk,按照安装文档反向操作进行卸载即可。
略。