ubuntu下安装zookeeper和kafka伪集群

原创

软件测试日常记录 2023-01-26 19:00:40 博主文章分类：ubuntu ©著作权

©著作权归作者所有：来自51CTO博客作者软件测试日常记录的原创作品，请联系作者获取转载授权，否则将追究法律责任

1.创建目录
mkdir -p /zookeeper/zkp-1 /zookeeper/zkp-2 /zookeeper/zkp-3
2.下载zookeeper
链接地址：http://mirror.bit.edu.cn/apache/zookeeper/ 命令下载：
cd /zookeeper/zkp-1
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz 3.解压
tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz
复制到zkp-2和zkp-3
cp -R /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin /zookeeper/zkp-2
cp -R /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin /zookeeper/zkp-3
4.修改配置文件
cd /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin/conf
vim zoo.cfg

tickTime=2000
dataDir= /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin/data
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.0.79:2888:3888
server.2=192.168.0.79:4888:5888
server.3=192.168.0.79:6888:7888

cd /zookeeper/zkp-2/apache-zookeeper-3.5.5-bin/conf
vim zoo.cfg

tickTime=2000
dataDir=/zookeeper/zkp-2/apache-zookeeper-3.5.5-bin/data
clientPort=2182
initLimit=5
syncLimit=2
server.1=192.168.0.79:2888:3888
server.2=192.168.0.79:4888:5888
server.3=192.168.0.79:6888:7888

cd /zookeeper/zkp-3/apache-zookeeper-3.5.5-bin/conf
vim zoo.cfg

tickTime=2000
dataDir=/zookeeper/zkp-3/apache-zookeeper-3.5.5-bin/data
clientPort=2183
initLimit=5
syncLimit=2
server.1=192.168.0.79:2888:3888
server.2=192.168.0.79:4888:5888
server.3=192.168.0.79:6888:7888

5.创建数据目录
mkdir -p /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin/data
mkdir -p /zookeeper/zkp-2/apache-zookeeper-3.5.5-bin/data
mkdir -p /zookeeper/zkp-3/apache-zookeeper-3.5.5-bin/data
6.写入myid文件
cd /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin/data
echo ‘1’ > myid
cd /zookeeper/zkp-2/apache-zookeeper-3.5.5-bin/data
echo ‘2’ > myid
cd /zookeeper/zkp-3/apache-zookeeper-3.5.5-bin/data
echo ‘3’ > myid
7.启动，三个都执行
cd /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin/bin
./zkServer.sh start
cd /zookeeper/zkp-2/apache-zookeeper-3.5.5-bin/bin
./zkServer.sh start
cd /zookeeper/zkp-3/apache-zookeeper-3.5.5-bin/bin
./zkServer.sh start

打印日志启动：./zkServer.sh start-foreground
8.查看jps结果
jps
8099 Jps
6808 QuorumPeerMain
6746 QuorumPeerMain
8014 QuorumPeerMain

kafka伪集群安装

1.创建目录
mkdir -p /kafka/kafka-1 /kafka/kafka-2 /kafka/kafka-3
2.下载kafka
链接:http://kafka.apache.org/downloads cd /kafka/kafka-1
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.1/kafka_2.12-2.2.1.tgz 3.解压
tar -zxvf kafka_2.12-2.2.1.tgz
复制到kafka-2和kafka-3
cp -R /kafka/kafka-1/kafka_2.12-2.2.1 /kafka/kafka-2/kafka_2.12-2.2.1
cp -R /kafka/kafka-1/kafka_2.12-2.2.1 /kafka/kafka-3/kafka_2.12-2.2.1
4.修改配置文件
cd /kafka/kafka-1/kafka_2.12-2.2.1/config
vim server.properties
(listeners=PLAINTEXT://wzj-ThinkPad-Edge-E440:9092//这个参数非常重要，想要外网访问docker内的集群，只能配置主机名，不能配置IP地址，命令hostname, 配置文件时注意参数broker.id,log.dirs,zookeeper.connect)

broker.id=1
listeners=PLAINTEXT://192.168.0.79:9092   
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/kafka/kafka-1/kafka_2.12-2.2.1/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.0.79:2181,192.168.0.79:2182,192.168.0.79:2183
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 可删除topic
delete.topic.enable=true
# 每间隔1秒钟时间，刷数据到磁盘
log.flush.interval.ms=1000
# 保留三天，也可以更短 
log.retention.hours=72

cd /kafka/kafka-2/kafka_2.12-2.2.1/config
vim server.properties

broker.id=2
listeners=PLAINTEXT://192.168.0.79:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/kafka/kafka-2/kafka_2.12-2.2.1/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.0.79:2181,192.168.0.79:2182,192.168.0.79:2183
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 可删除topic
delete.topic.enable=true
# 每间隔1秒钟时间，刷数据到磁盘
log.flush.interval.ms=1000
# 保留三天，也可以更短 
log.retention.hours=72

cd /kafka/kafka-3/kafka_2.12-2.2.1/config
vim server.properties

broker.id=3
listeners=PLAINTEXT://192.168.0.79:9094
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/kafka/kafka-3/kafka_2.12-2.2.1/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.0.79:2181,192.168.0.79:2182,192.168.0.79:2183
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 可删除topic
delete.topic.enable=true
# 每间隔1秒钟时间，刷数据到磁盘
log.flush.interval.ms=1000
# 保留三天，也可以更短 
log.retention.hours=72

4.启动，三个都执行
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-server-start.sh /kafka/kafka-1/kafka_2.12-2.2.1/config/server.properties &

cd /kafka/kafka-2/kafka_2.12-2.2.1
bin/kafka-server-start.sh /kafka/kafka-2/kafka_2.12-2.2.1/config/server.properties &

cd /kafka/kafka-3/kafka_2.12-2.2.1
bin/kafka-server-start.sh /kafka/kafka-3/kafka_2.12-2.2.1/config/server.properties &

后台启动方式:
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-server-start.sh -daemon /kafka/kafka-1/kafka_2.12-2.2.1/config/server.properties

cd /kafka/kafka-2/kafka_2.12-2.2.1
bin/kafka-server-start.sh -daemon /kafka/kafka-2/kafka_2.12-2.2.1/config/server.properties

cd /kafka/kafka-3/kafka_2.12-2.2.1
bin/kafka-server-start.sh -daemon /kafka/kafka-3/kafka_2.12-2.2.1/config/server.properties

netstat lntp|grep 9092
5.测试kafka集群
#创建topic
cd /kafka/kafka-1/kafka_2.12-2.2.1

bin/kafka-topics.sh --create --zookeeper 192.168.0.79:2181,192.168.0.79:3181,192.168.0.79:4181 --replication-factor 3 --partitions 3 --topic test
#显示topic信息
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-topics.sh --describe --zookeeper 192.168.0.79:2181,192.168.0.79:3181,192.168.0.79:4181 --topic test
#列出topic
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-topics.sh --list --zookeeper 192.168.0.79:2181,192.168.0.79:3181,192.168.0.79:4181
#创建 producer
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-console-producer.sh --broker-list master:9092 -topic test
#创建 consumer
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-console-consumer.sh --zookeeper 192.168.0.79:2181,192.168.0.79:3181,192.168.0.79:4181 -topic test --from-beginning
#删除topic
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-topics.sh --delete --zookeeper 192.168.0.79:2181,192.168.0.79:3181,192.168.0.79:4181 --topic test
#查询topic内容
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topicName --from-beginning
6.关闭kafka
cd /kafka/kafka-1/kafka_2.12-2.2.1
bin/kafka-server-stop.sh /kafka/kafka-1/kafka_2.12-2.2.1/config/server.properties

cd /kafka/kafka-2/kafka_2.12-2.2.1
bin/kafka-server-stop.sh /kafka/kafka-2/kafka_2.12-2.2.1/config/server.properties

cd /kafka/kafka-3/kafka_2.12-2.2.1
bin/kafka-server-stop.sh /kafka/kafka-3/kafka_2.12-2.2.1/config/server.properties
7.关闭zookeeper
cd /zookeeper/zkp-1/apache-zookeeper-3.5.5-bin
bin/zkServer.sh stop conf/zoo.cfg

cd /zookeeper/zkp-2/apache-zookeeper-3.5.5-bin
bin/zkServer.sh stop conf/zoo.cfg

cd /zookeeper/zkp-3/apache-zookeeper-3.5.5-bin
bin/zkServer.sh stop conf/zoo.cfg