Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十七）：kafka manager安装

转载

mob60475702efd6 2018-09-11 10:59:00

文章标签 Kafka kafka spark scala hadoop 文章分类 后端开发

一、kafka-manager简介

为了简化开发者和服务工程师维护Kafka集群的工作，yahoo构建了一个叫做Kafka管理器的基于Web工具，叫做 Kafka Manager。这个管理工具可以很容易地发现分布在集群中的哪些topic分布不均匀，或者是分区在整个集群分布不均匀的的情况。它支持管理多个集群、选择副本、副本重新分配以及创建Topic。同时，这个管理工具也是一个非常好的可以快速浏览这个集群的工具，有如下功能：

管理多个集群
轻松检查群集状态（主题，消费者，偏移，代理，副本分发，分区分发）
运行首选副本选举
使用选项生成分区分配以选择要使用的代理
运行分区重新分配（基于生成的分配）
使用可选主题配置创建主题（0.8.1.1具有与0.8.2+不同的配置）
删除主题（仅支持0.8.2+并记住在代理配置中设置delete.topic.enable = true）
主题列表现在指示标记为删除的主题（仅支持0.8.2+）
批量生成多个主题的分区分配，并可选择要使用的代理
批量运行重新分配多个主题的分区
将分区添加到现有主题
更新现有主题的配置
kafka-manager 项目地址：https://github.com/yahoo/kafka-manager/

二、kafka-manager安装

1.下载安装包

使用Git或者直接从Releases中下载，这里下载 1.3.3.18 版本：https://github.com/yahoo/kafka-manager/releases

wget https://github.com/yahoo/kafka-manager/archive/1.3.3.18.zip

2.解压安装包

Last login: Thu Sep  6 20:51:48 2018 from 192.168.0.103
[spark@master ~]$ cd /opt/
[spark@master opt]$ wget https://github.com/yahoo/kafka-manager/archive/1.3.3.18.zip
[spark@master opt]$ ll
total 1076
drwxr-xr-x.  6 root   root        99 Jul  5 07:00 apache-maven-3.5.4
drwxr-xr-x.  9 spark  spark      155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark     4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark      172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root       153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143     255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root       101 Jul 17 07:01 kafka_2.11-1.1.0
-rw-r--r--.  1 root   root   1075857 Sep 11 21:32 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark      232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root      9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark      252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark       67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark     4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000      50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop     236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark     4096 Aug  8 00:23 zookeeper-3.4.12
[spark@master opt]$ su root
Password: 
[root@master opt]# unzip kafka-manager-1.3.3.18.zip 
Archive:  kafka-manager-1.3.3.18.zip
8dcdbf8fabb0001691c9b52b447b656f498b4d7b
   creating: kafka-manager-1.3.3.18/
   。。。。
  inflating: kafka-manager-1.3.3.18/test/kafka/test/SeededBroker.scala  
[root@master opt]# ls
apache-maven-3.5.4         jdk1.8.0_171                mysql57-community-release-el7-8.noarch.rpm  spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2        kafka_2.11-1.1.0            nifi-1.7.1                                  zookeeper-3.4.12
elasticsearch-head-master  kafka-manager-1.3.3.18      node-8.9.1
hadoop-2.9.0               kafka-manager-1.3.3.18.zip  node-v8.9.1
hdfs-over-ftp-master       kibana-6.2.2-linux-x86_64   scala-2.11.0

3.sbt编译

1）yum安装sbt(因为kafka-manager需要sbt编译)

[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ls
app  build.sbt  conf  img  LICENCE  project  public  README.md  sbt  src  test
[root@master kafka-manager-1.3.3.18]# sbt
bash: sbt: command not found
[root@master kafka-manager-1.3.3.18]# cd ..
[root@master opt]#  curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   160    0   160    0     0     21      0 --:--:--  0:00:07 --:--:--    40
[root@master opt]# ll
total 1080
drwxr-xr-x.  6 root   root        99 Jul  5 07:00 apache-maven-3.5.4
-rw-r--r--.  1 root   root       160 Sep 11 21:38 bintray-sbt-rpm.repo
drwxr-xr-x.  9 spark  spark      155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark     4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark      172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root       153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143     255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root       101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x.  9 root   root       189 Jul  7 04:44 kafka-manager-1.3.3.18
-rw-r--r--.  1 root   root   1075857 Sep 11 21:32 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark      232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root      9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark      252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark       67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark     4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000      50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop     236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark     4096 Aug  8 00:23 zookeeper-3.4.12
[root@master opt]# mv bintray-sbt-rpm.repo /etc/yum.repos.d/
[root@master opt]#  yum install sbt
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirrors.zju.edu.cn
 * extras: mirrors.shu.edu.cn
 * updates: mirrors.zju.edu.cn
base                                                                                                         | 3.6 kB  00:00:00     
bintray--sbt-rpm                                                                                             | 1.3 kB  00:00:00     
extras                                                                                                       | 3.4 kB  00:00:00     
mysql-connectors-community                                                                                   | 2.5 kB  00:00:00     
mysql-tools-community                                                                                        | 2.5 kB  00:00:00     
mysql57-community                                                                                            | 2.5 kB  00:00:00     
updates                                                                                                      | 3.4 kB  00:00:00     
(1/4): extras/7/x86_64/primary_db                                                                            | 187 kB  00:00:00     
(2/4): mysql-connectors-community/x86_64/primary_db                                                          |  26 kB  00:00:00     
(3/4): bintray--sbt-rpm/primary                                                                              | 3.8 kB  00:00:02     
(4/4): updates/7/x86_64/primary_db                                                                           | 5.2 MB  00:00:02     
bintray--sbt-rpm                                                                                                              32/32
Resolving Dependencies
--> Running transaction check
---> Package sbt.noarch 0:1.2.1-0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================
 Package                   Arch                         Version                        Repository                              Size
====================================================================================================================================
Installing:
 sbt                       noarch                       1.2.1-0                        bintray--sbt-rpm                       1.1 M

Transaction Summary
====================================================================================================================================
Install  1 Package

Total download size: 1.1 M
Installed size: 1.2 M
Is this ok [y/d/N]: y
Downloading packages:
sbt-1.2.1.rpm                                                                                                | 1.1 MB  00:00:19     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : sbt-1.2.1-0.noarch                                                                                               1/1 
  Verifying  : sbt-1.2.1-0.noarch                                                                                               1/1 

Installed:
  sbt.noarch 0:1.2.1-0                                                                                                              

Complete!

改仓库地址：（sbt 默认下载库文件很慢, 还时不时被打断），我们可以在用户目录下创建 touch ~/.sbt/repositories, 填上阿里云的镜像 # vi ~/.sbt/repositories

[root@master opt]# ls
apache-maven-3.5.4         jdk1.8.0_171                mysql57-community-release-el7-8.noarch.rpm  spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2        kafka_2.11-1.1.0            nifi-1.7.1                                  zookeeper-3.4.12
elasticsearch-head-master  kafka-manager-1.3.3.18      node-8.9.1
hadoop-2.9.0               kafka-manager-1.3.3.18.zip  node-v8.9.1
hdfs-over-ftp-master       kibana-6.2.2-linux-x86_64   scala-2.11.0
[root@master opt]# sbt
Getting org.scala-sbt sbt 1.2.1  (this may take some time)...
^C[root@master opt]# sbt -version
Getting org.scala-sbt sbt 1.2.1  (this may take some time)...
^C[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ./sbt clean dist
Downloading sbt launcher for 0.13.9:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar
    To  /root/.sbt/launchers/0.13.9/sbt-launch.jar
Download failed. Obtain the jar manually and place it at /root/.sbt/launchers/0.13.9/sbt-launch.jar
[root@master kafka-manager-1.3.3.18]# vi ~/.sbt/repositories

[repositories]
  local
  aliyun-nexus: http://maven.aliyun.com/nexus/content/groups/public/
  jcenter: https://jcenter.bintray.com/
  typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[c
lassifier]).[ext], bootOnly
  maven-central
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~

2）编译kafka-manager

[root@master kafka-manager-1.3.3.18]# ./sbt clean dist
Downloading sbt launcher for 0.13.9:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar
    To  /root/.sbt/launchers/0.13.9/sbt-launch.jar
[info] Loading project definition from /opt/kafka-manager-1.3.3.18/project
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[info] Set current project to kafka-manager (in build file:/opt/kafka-manager-1.3.3.18/)
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[warn] Credentials file /root/.bintray/.credentials does not exist
[success] Total time: 0 s, completed Sep 11, 2018 10:54:50 PM
。。。。。
[info]  [SUCCESSFUL ] jline#jline;2.12.1!jline.jar (340ms)
[info] Done updating.
[warn] Scala version was updated by one of library dependencies:
[warn]  * org.scala-lang:scala-library:(2.11.6, 2.11.8, 2.11.11, 2.11.7, 2.11.5, 2.11.0) -> 2.11.12
[warn] To force scalaVersion, add the following:
[warn]  ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) }
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * org.webjars:jquery:1.11.1 -> 2.1.4
[warn] Run 'evicted' to see detailed eviction warnings
[info] Wrote /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18.pom
[info] Compiling 127 Scala sources and 2 Java sources to /opt/kafka-manager-1.3.3.18/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info] Main Scala API documentation to /opt/kafka-manager-1.3.3.18/target/scala-2.11/api...
[info]   Compilation completed in 8.323 s
model contains 701 documentable templates
[info] Main Scala API documentation successful.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-javadoc.jar ...
[info] Done packaging.
[info] LESS compiling on 1 source(s)
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-web-assets.jar ...
[info] Done packaging.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18.jar ...
[info] Done packaging.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-sans-externalized.jar ...
[info] Done packaging.
[info] 
[info] Your package is ready in /opt/kafka-manager-1.3.3.18/target/universal/kafka-manager-1.3.3.18.zip
[info] 
[success] Total time: 376 s, completed Sep 11, 2018 11:01:06 PM
[root@master kafka-manager-1.3.3.18]# ls
app  build.sbt  conf  img  LICENCE  project  public  README.md  sbt  src  target  test
[root@master kafka-manager-1.3.3.18]# cd target/universal
[root@master universal]# ls
kafka-manager-1.3.3.18.zip  scripts
[root@master universal]# scp kafka-manager-1.3.3.18.zip /opt/kafka-manager-1.3.3.18.zip

4.安装

重新解压编译好的kafka-manager-1.3.3.18.zip&修改配置文件

[root@master universal]# cd /opt/
[root@master opt]# ls
apache-maven-3.5.4         hadoop-2.9.0          kafka_2.11-1.1.0            kibana-6.2.2-linux-x86_64                   node-8.9.1    spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2        hdfs-over-ftp-master  kafka-manager-1.3.3.18      mysql57-community-release-el7-8.noarch.rpm  node-v8.9.1   zookeeper-3.4.12
elasticsearch-head-master  jdk1.8.0_171          kafka-manager-1.3.3.18.zip  nifi-1.7.1                                  scala-2.11.0
[root@master opt]# mv kafka-manager-1.3.3.18 kafka-manager-1.3.3.18-source
[root@master opt]# ll
total 77696
drwxr-xr-x.  6 root   root         99 Jul  5 07:00 apache-maven-3.5.4
drwxr-xr-x.  9 spark  spark       155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark      4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark       172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root        153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143      255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root        101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x. 10 root   root        203 Sep 11 22:54 kafka-manager-1.3.3.18-source
-rw-r--r--.  1 root   root   79534088 Sep 11 23:01 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark       232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root       9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark       252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark        67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark      4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000       50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop      236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark      4096 Aug  8 00:23 zookeeper-3.4.12
[root@master opt]# unzip kafka-manager-1.3.3.18.zip 
Archive:  kafka-manager-1.3.3.18.zip
  inflating: kafka-manager-1.3.3.18/lib/kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar  
  。。。。
  inflating: kafka-manager-1.3.3.18/share/doc/api/index/index-d.html  
  inflating: kafka-manager-1.3.3.18/README.md  
[root@master opt]# ll
total 77696
drwxr-xr-x.  6 root   root         99 Jul  5 07:00 apache-maven-3.5.4
drwxr-xr-x.  9 spark  spark       155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x.  7 spark  spark      4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark  spark       172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x.  4 root   root        153 Jul  9 05:35 hdfs-over-ftp-master
drwxr-xr-x.  8     10    143      255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx.  7 root   root        101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x.  6 root   root         70 Sep 11 23:02 kafka-manager-1.3.3.18
drwxr-xr-x. 10 root   root        203 Sep 11 22:54 kafka-manager-1.3.3.18-source
-rw-r--r--.  1 root   root   79534088 Sep 11 23:01 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark  spark       232 Feb 17  2018 kibana-6.2.2-linux-x86_64
-rw-r--r--.  1 root   root       9116 Apr 11  2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark  spark       252 Aug  6 20:23 nifi-1.7.1
drwxr-xr-x.  7 spark  spark        67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark  spark      4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x.  6   2000   2000       50 Apr 17  2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop      236 Sep  6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark  spark      4096 Aug  8 00:23 zookeeper-3.4.12
[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ls
bin  conf  lib  README.md  share
[root@master kafka-manager-1.3.3.18]# cd conf/
[root@master conf]# ls
application.conf  consumer.properties  logback.xml  logger.xml  routes
[root@master conf]# vim application.conf 

# Secret key
# ~~~~~
# The secret key is used to secure cryptographics functions.
# If you deploy your application to several instances be sure to use the same key!
play.crypto.secret="^<csmm5Fx4d=r2HEX8pelM3iBkFVv?k[mc;IZE<_Qoq8EkX_/7@Zt6dP05Pzea3U"
play.crypto.secret=${?APPLICATION_SECRET}

# The application languages
# ~~~~~
play.i18n.langs=["en"]

play.http.requestHandler = "play.http.DefaultHttpRequestHandler"
play.http.context = "/"
play.application.loader=loader.KafkaManagerLoader

#kafka-manager.zkhosts="kafka-manager-zookeeper:2181"
kafka-manager.zkhosts="192.168.0.120:2181,192.168.0.121:2181,192.168.0.122:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
pinned-dispatcher.type="PinnedDispatcher"
pinned-dispatcher.executor="thread-pool-executor"
application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFea
ture"]

akka {
  loggers = ["akka.event.slf4j.Slf4jLogger"]
  loglevel = "INFO"
}

akka.logger-startup-timeout = 60s

basicAuthentication.enabled=false
basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED}
basicAuthentication.username="admin"
basicAuthentication.username=${?KAFKA_MANAGER_USERNAME}
basicAuthentication.password="password"
basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}
basicAuthentication.realm="Kafka-Manager"
basicAuthentication.excluded=["/api/health"] # ping the health of your instance without authentification

kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE}
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
~                                                                                                                                   
"application.conf" 46L, 1682C written

5.启动服务

启动zk集群，kafka集群，再启动kafka-manager服务。

bin/kafka-manager 默认的端口是9000，可通过 -Dhttp.port，指定端口; -Dconfig.file=conf/application.conf指定配置文件:

[root@master conf]# cd ..
[root@master kafka-manager-1.3.3.18]# ls
bin  conf  lib  README.md  share
[root@master kafka-manager-1.3.3.18]# 
[root@master kafka-manager-1.3.3.18]# nohup bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=19093 &

jps查看：

[root@master spark]# jps
4596 QuorumPeerMain
4956 Kafka
4476 ProdServerStart
5116 Jps
[root@master spark]#

WebUI查看：http://192.168.0.120:19093/ 出现如下界面则启动成功。

其他就是在ui上新建cluster，新建topic，查看topic等操作。

基础才是编程人员应该深入研究的问题，比如:

1）List/Set/Map内部组成原理|区别

2）mysql索引存储结构&如何调优/b-tree特点、计算复杂度及影响复杂度的因素。。。

3）JVM运行组成与原理及调优

4）Java类加载器运行原理

5）Java中GC过程原理|使用的回收算法原理

6）Redis中hash一致性实现及与hash其他区别

7）Java多线程、线程池开发、管理Lock与Synchroined区别

8）Spring IOC/AOP 原理；加载过程的。。。

【+加关注】。

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：C# 多线程（转）

下一篇：x64位windows上程序开发的注意事项

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯