centos搭建kafka集群 linux安装kafka集群

  大家好呀，今天给大家带来的是，最新版kafka集群的安装教程，希望给小伙伴们一点小小的帮助。

注意：提前安装好jdk, Jdk安装教程

1、准备安装包，Kafka官网下载

2、kafka安装需要zk,注意：kafka安装版本和zk版本要一致，下载以后，先解压kafka，进入libs，查看依赖的zk版本再下载,切记切记！！！zookeeper官网下载

[root@VM-4-2-centos jdk]# java -version
java version "1.8.0_333"
Java(TM) SE Runtime Environment (build 1.8.0_333-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.333-b02, mixed mode)

3、先解压zk，然后重命名,同时配置环境变量

[root@VM-4-2-centos kafka]# tar -zxvf apache-zookeeper-3.6.3-bin.tar.gz 
[root@VM-4-2-centos kafka]# mv apache-zookeeper-3.6.3-bin/ zookeeper-3.6.3

echo 'export ZOOKEEPER_HOME=/opt/kafka/zookeeper/zookeeper-3.6.3' >> /etc/profile
echo 'export PATH=$PATH:$ZOOKEEPER_HOME/bin' >> /etc/profile
source /etc/profile

3A、新建一个zookeeper文件夹，然后移进去

[root@VM-4-2-centos kafka]# mkdir zookeeper
[root@VM-4-2-centos kafka]# mv zookeeper-3.6.3 ./zookeeper/

5、修改conf文件夹配置

[root@VM-4-2-centos kafka]# cd zookeeper/zookeeper-3.6.3/conf
[root@VM-4-2-centos zookeeper-3.6.3]# mv zoo_sample.cfg zoo.cfg 
[root@VM-4-2-centos conf]# vi zoo.cfg

注意:server 设置broker Id的服务地址，本机ip一定要用0.0.0.0代替

server.1=0.0.0.0:2888:3888，1是我们data/myid,2888是服务端通信端口，3888服务端之间选举端口，云服务器开放2181 2888 3888端口

dataDir=/opt/kafka/zookeeper/zookeeper-3.6.3/data

dataLogDir=/opt/kafka/zookeeper/zookeeper-3.6.3/log

server.1=0.0.0.0:2888:3888
 
server.2=tencent02:2888:3888
 
server.3=tencent03:2888:3888

6、创建data 、log文件夹，然后进入data,
添加内容：1 ，其他两台主机分别配置 2和3

[root@VM-4-2-centos conf]# cd ..
[root@VM-4-2-centos zookeeper-3.6.3]# mkdir data && mkdir log
[root@VM-12-13-centos zookeeper-3.6.3]# cd data

echo 1 > myid

7、上去，然后启动，第二台、第三台服务器，请重复3-7步骤

[root@VM-4-2-centos data]# cd ..
[root@VM-4-2-centos zookeeper-3.6.3]# ./bin/zkServer.sh start

8、查看zk是否启动成功，没有问题

[root@VM-4-2-centos zookeeper-3.6.3]# jps
92838 QuorumPeerMain
94502 Jps
[root@VM-4-2-centos zookeeper-3.6.3]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1741/sshd           
tcp6       0      0 :::2181                 :::*                    LISTEN      92838/java          
tcp6       0      0 :::3888                 :::*                    LISTEN      92838/java          
tcp6       0      0 :::8080                 :::*                    LISTEN      92838/java          
tcp6       0      0 :::39381                :::*                    LISTEN      92838/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1741/sshd

9、查看zk集群状态

[root@VM-12-13-centos logs]# cd /opt/kafka/zookeeper/zookeeper-3.6.3
[root@VM-12-13-centos zookeeper-3.6.3]# ./bin/zkServer.sh status ./conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: ./conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

切换到第二台服务器：

[root@VM-4-12-centos kafka]# cd /opt/kafka/zookeeper/zookeeper-3.6.3
[root@VM-4-12-centos zookeeper-3.6.3]# ./bin/zkServer.sh status ./conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: ./conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

10、接下来是正式环节，搭建kafka集群

[root@VM-4-2-centos zookeeper-3.6.3]# cd /opt/kafka/
[root@VM-4-2-centos kafka]# ll
total 96716
-rw-r--r-- 1 root root 12516362 May 31 17:33 apache-zookeeper-3.6.3-bin.tar.gz
-rw-r--r-- 1 root root 86512293 Apr 10 00:46 kafka_2.13-3.0.1.tgz
drwxr-xr-x 3 root root     4096 Jun  1 07:39 zookeeper

11、解压kafka,然后重命名

[root@VM-4-2-centos kafka]# tar -zxvf kafka_2.13-3.0.1.tgz 
[root@VM-4-2-centos kafka]# mv kafka_2.13-3.0.1 kafka

12、进入kakfa,创建log文件夹

[root@VM-4-2-centos kafka]# cd kafka/
[root@VM-4-12-centos kafka]# mkdir log
[root@VM-4-12-centos kafka]# ll
total 68
drwxr-xr-x 3 root root  4096 Mar  4 00:20 bin
drwxr-xr-x 3 root root  4096 Mar  4 00:20 config
drwxr-xr-x 2 root root  4096 Jun  1 08:26 libs
-rw-r--r-- 1 root root 14587 Mar  4 00:15 LICENSE
drwxr-xr-x 2 root root  4096 Mar  4 00:20 licenses
drwxr-xr-x 2 root root  4096 Jun  1 08:34 log
-rw-r--r-- 1 root root 28184 Mar  4 00:15 NOTICE
drwxr-xr-x 2 root root  4096 Mar  4 00:20 site-docs

13、修改kafka配置，broker.id 不同服务器 分别为1 、2 、3

[root@VM-4-12-centos kafka]# vi config/server.properties

broker.id=1

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://101.35.245.191:9092

log.dirs=/opt/kafka/kafka/log

# topic 在当前broker上的分片个数，与broker保持一致
num.partitions=3

# 设置zookeeper集群地址与端口如下：
zookeeper.connect=tencent01:2181,tencent02:2181,tencent03:2181

14、启动kafka集群

[root@VM-4-12-centos kafka]# ./bin/kafka-server-start.sh -daemon ./config/server.properties &

15、查看，没有问题

[root@VM-4-2-centos kafka]# jps  
112519 Jps
112361 Kafka
109642 QuorumPeerMain

[root@VM-4-12-centos kafka]# lsof -i:9092
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    32026 root  134u  IPv6 180770      0t0  TCP *:XmlIpcRegSvc (LISTEN)
java    32026 root  153u  IPv6 182221      0t0  TCP VM-4-12-centos:nimhub->tencent02:XmlIpcRegSvc (ESTABLISHED)
java    32026 root  154u  IPv6 180774      0t0  TCP VM-4-12-centos:XmlIpcRegSvc->tencent02:nimhub (ESTABLISHED)
java    32026 root  158u  IPv6 181065      0t0  TCP VM-4-12-centos:51164->tencent01:XmlIpcRegSvc (ESTABLISHED)

16、然后我们连接上kafka，新建一个topic
–replication-factor 2 两个副本
–partitions 1 一个分区
–topic test 一个test主题

[root@VM-4-2-centos kafka]# cd /opt/kafka/kafka/
[root@VM-4-2-centos kafka]# ./bin/kafka-topics.sh --create --replication-factor 2 --partitions 1 --topic test --bootstrap-server 101.35.245.191:9092

17、切换到第二台服务器，查看主题是否创建成功

[root@VM-4-12-centos zookeeper-3.6.3]# cd /opt/kafka/kafka/
[root@VM-4-12-centos kafka]# ./bin/kafka-topics.sh --list --bootstrap-server 101.35.245.191:9092
test

18、启动生产者

[root@VM-4-2-centos kafka]# ./bin/kafka-console-producer.sh --broker-list tencent01:9092 --topic test

19、启动消费者，第二台和第三台服务器

./bin/kafka-console-consumer.sh --bootstrap-server tencent02:9092 --topic test

./bin/kafka-console-consumer.sh --bootstrap-server tencent03:9092 --topic test

20、生产者发送消息，接收者接收消息，成功！

21、至此，整个kafka集群搭建完毕