Kafka JMX 监控 之 jmxtrans + influxdb + grafana

 

目录

效果图

环境准备

安装 influxdb

安装我们刚刚下载 influxdb rpm文件

查看默认配置

修改参数

启动 influxdb

查看启动状态

设置基本配置

 influxdb 其他命令扩展

安装 jmxtrans

可能遇到的异常

验证jmxtrans是否成功运行

安装 Grafana

 安装

influxDB 与 Grafana 监控模板

 

---

效果图

先看下

环境准备

JDK：1.8

jmxtrans 安装包

# 我们系统是CentOS，这里选择rpm
https://github.com/downloads/jmxtrans/jmxtrans/jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm
# 其他系统在下面链接中可以找到
https://github.com/jmxtrans/jmxtrans/downloads

# 下载源码，后面编译使用
https://github.com/jmxtrans/jmxtrans/releases
# 我这里使用的271版本
https://github.com/jmxtrans/jmxtrans/archive/jmxtrans-parent-271.tar.gz

influxdb 安装包

# 下载安装包
https://dl.influxdata.com/influxdb/releases/influxdb-1.8.0.x86_64.rpm
# 官网下载页面
https://portal.influxdata.com/downloads/

Grafana 安装包

https://dl.grafana.com/oss/release/grafana-6.7.3-1.x86_64.rpm

安装 influxdb

安装我们刚刚下载 influxdb rpm文件

rpm -ivh influxdb-1.8.0.x86_64.rpm

查看默认配置

> influxd config

Merging with configuration at: /etc/influxdb/influxdb.conf
reporting-disabled = false
bind-address = "127.0.0.1:8088"

[meta]
  dir = "/var/lib/influxdb/meta"
  retention-autocreate = true
  logging-enabled = true

[data]
  dir = "/var/lib/influxdb/data"
  index-version = "inmem"
  wal-dir = "/var/lib/influxdb/wal"
  wal-fsync-delay = "0s"
  validate-keys = false
  query-log-enabled = true
  cache-max-memory-size = 1073741824
  cache-snapshot-memory-size = 26214400
  cache-snapshot-write-cold-duration = "10m0s"
  compact-full-write-cold-duration = "4h0m0s"
  compact-throughput = 50331648
  compact-throughput-burst = 50331648
  max-series-per-database = 1000000
  max-values-per-tag = 100000
  max-concurrent-compactions = 0
  max-index-log-file-size = 1048576
  series-id-set-cache-size = 100
  series-file-max-concurrent-snapshot-compactions = 0
  trace-logging-enabled = false
  tsm-use-madv-willneed = false

[coordinator]
  write-timeout = "10s"
  max-concurrent-queries = 0
  query-timeout = "0s"
  log-queries-after = "0s"
  max-select-point = 0
  max-select-series = 0
  max-select-buckets = 0

[retention]
  enabled = true
  check-interval = "30m0s"

[shard-precreation]
  enabled = true
  check-interval = "10m0s"
  advance-period = "30m0s"

[monitor]
  store-enabled = true
  store-database = "_internal"
  store-interval = "10s"

[subscriber]
  enabled = true
  http-timeout = "30s"
  insecure-skip-verify = false
  ca-certs = ""
  write-concurrency = 40
  write-buffer-size = 1000

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = true
  suppress-write-log = false
  write-tracing = false
  flux-enabled = false
  flux-log-enabled = false
  pprof-enabled = true
  pprof-auth-enabled = false
  debug-pprof-enabled = false
  ping-auth-enabled = false
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"
  https-private-key = ""
  max-row-limit = 0
  max-connection-limit = 0
  shared-secret = ""
  realm = "InfluxDB"
  unix-socket-enabled = false
  unix-socket-permissions = "0777"
  bind-socket = "/var/run/influxdb.sock"
  max-body-size = 25000000
  access-log-path = ""
  max-concurrent-write-limit = 0
  max-enqueued-write-limit = 0
  enqueued-write-timeout = 30000000000

[logging]
  format = "auto"
  level = "info"
  suppress-logo = false

[[graphite]]
  enabled = false
  bind-address = ":2003"
  database = "graphite"
  retention-policy = ""
  protocol = "tcp"
  batch-size = 5000
  batch-pending = 10
  batch-timeout = "1s"
  consistency-level = "one"
  separator = "."
  udp-read-buffer = 0

[[collectd]]
  enabled = false
  bind-address = ":25826"
  database = "collectd"
  retention-policy = ""
  batch-size = 5000
  batch-pending = 10
  batch-timeout = "10s"
  read-buffer = 0
  typesdb = "/usr/share/collectd/types.db"
  security-level = "none"
  auth-file = "/etc/collectd/auth_file"
  parse-multivalue-plugin = "split"

[[opentsdb]]
  enabled = false
  bind-address = ":4242"
  database = "opentsdb"
  retention-policy = ""
  consistency-level = "one"
  tls-enabled = false
  certificate = "/etc/ssl/influxdb.pem"
  batch-size = 1000
  batch-pending = 5
  batch-timeout = "1s"
  log-point-errors = true

[[udp]]
  enabled = false
  bind-address = ":8089"
  database = "udp"
  retention-policy = ""
  batch-size = 5000
  batch-pending = 10
  read-buffer = 0
  batch-timeout = "1s"
  precision = ""

[continuous_queries]
  log-enabled = true
  enabled = true
  query-stats-enabled = false
  run-interval = "1s"

[tls]
  min-version = ""
  max-version = ""

修改参数

默认influxDB使用以下端口

• 8086: 用于客户端和服务端交互的HTTP API
• 8088: 用于提供备份和恢复的RPC服务

我这里修改配置文件,使用8087端口，我这里8088和其他服务冲突了

同时修改了数据保存的路径

> vim /etc/influxdb/influxdb.conf

bind-address = "127.0.0.1:8087"
# metadata 保存路径
dir = "/root/jast/influxdb/meta"
#数据保存路径
dir = "/root/jast/influxdb/data"
#`write-ahead-log(WAL)保存路径
wal-dir = "/root/jast/influxdb/wal"

 注意：选择的路径要有权限，否则会启动失败

启动 influxdb

systemctl start influxdb

查看启动状态

systemctl status influxdb

 

此时influxdb已经启动成功

我们也可以指定配置文件启动，在/etc/influxdb/influxdb.conf ,是默认的目录也可以不指定

influxd -config /etc/influxdb/influxdb.conf

设置基本配置

在服务器 输入 influx 进入交互页面

[root@ecs-t-001-0001 influx]# influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
>

 创建用户

CREATE USER "admin" WITH PASSWORD '123456' WITH ALL PRIVILEGES

 创建数据库（后面保存数据用）

create database "jmxDB"

查看是否创建成功

[root@ecs-t-001-0001 jmxtrans]# influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
> show databases
name: databases
name
----
_internal
jmxDB
>

 influxdb 其他命令扩展

#创建数据库
create database "db_name"
#显示所有的数据库
show databases
#删除数据库
drop database "db_name"
#使用数据库
use db_name
#显示该数据库中所有的表
show measurements
#创建表，直接在插入数据的时候指定表名
insert test,host=127.0.0.1,monitor_name=test count=1
#删除表
drop measurement "measurement_name"
#退出
quit

 

安装 jmxtrans

安装我们刚刚下载jmxtrans rpm文件

rpm -ivh jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm

安装完成后默认安装目录在

[root@ecs-t-001-0001 jmxtrans]# whereis jmxtrans
jmxtrans: /usr/share/jmxtrans

这里我们先简单配置Kafka 的Memory 监控，其他配置在文末统一整理

我们创建json文件供jmxtrans读取，json文件名称自己根据业务取名即可

{
  "servers" : [ {
    "port" : "9393",
    "host" : "172.11.0.1",
    "queries" : [ {
      "obj" : "java.lang:type=Memory",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "resultAlias":"jvmMemory",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://172.11.0.1:8086/",
        "username" : "admin",
        "password" : "123456",
        "database" : "jmxDB",
        "tags"     : {"application" : "kafka_server"}
      } ]
    } ]
  } ]
}

简单解释一下上面的说明

port: 我们要监控的Kafka JMX端口
host：我们要监控的Kafka host
resultAlias：自定义表名，收集到的数据会存入influxdb的定义的表中，自动创建
outputWriters为连接influxdb的配置
@class不需要修改
url：influxdb的机器+端口，默认端口8086
username和password：influxdb的用户和密码
database：influxdb数据库（我们刚刚创建的）

启动之前我们把 /usr/share/jmxtrans 目录下的所有 .json 文件换个名，因为它会默认会读取 /usr/share/jmxtrans 目录下的所有json文件

在 /usr/share/jmxtrans 目录下启动 jmxtrans.sh

jmxtrans.sh start

到这里正常来说就是要启动成功了，我们先说下可能遇到的异常

可能遇到的异常

异常1

[root@ecs-t-001-0001 jmxtrans]# Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=384m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
MaxTenuringThreshold of 16 is invalid; must be between 0 and 15
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

MaxTenuringThreshold 这个参数用于控制对象能经历多少次Minor GC才晋升到旧生代

提示设置的是16，但是范围在0-15，我们直接修改一下启动脚本 jmxtrans.sh

> vim jmxtrans.sh

GC_OPTS=${GC_OPTS:-"-Xms${HEAP_SIZE}M -Xmx${HEAP_SIZE}M -XX:+UseConcMarkSweepGC -XX:NewRatio=${NEW_RATIO} -XX:NewSize=${NEW_SIZE}m -XX:MaxNewSize=${NEW_SIZE}m -XX:MaxTenuringThreshold=15 -XX:GCTimeRatio=9 -XX:PermSize=${PERM_SIZE}m -XX:MaxPermSize=${MAX_PERM_SIZE}m -XX:+UseTLAB -XX:CMSInitiatingOccupancyFraction=${IO_FRACTION} -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=${CPU_CORES} -Dsun.rmi.dgc.server.gcInterval=28800000 -Dsun.rmi.dgc.client.gcInterval=28800000"}

异常2

[root@ecs-t-001-0001 jmxtrans]# Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=384m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Exception in thread "main" com.googlecode.jmxtrans.util.LifecycleException: com.googlecode.jmxtrans.util.LifecycleException: Error parsing json: /var/lib/jmxtrans/kafka.json
	at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:146)
	at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
	at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
Caused by: com.googlecode.jmxtrans.util.LifecycleException: Error parsing json: /var/lib/jmxtrans/kafka.json
	at com.googlecode.jmxtrans.JmxTransformer.processFilesIntoServers(JmxTransformer.java:358)
	at com.googlecode.jmxtrans.JmxTransformer.startupSystem(JmxTransformer.java:301)
	at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:142)
	... 2 more
Caused by: java.lang.IllegalArgumentException: Invalid type id 'com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory' (for id type 'Id.class'): no such class found
	at org.codehaus.jackson.map.jsontype.impl.ClassNameIdResolver.typeFromId(ClassNameIdResolver.java:89)
	at org.codehaus.jackson.map.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:73)
	at org.codehaus.jackson.map.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:65)
	at org.codehaus.jackson.map.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:81)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:118)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
	at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:149)
	at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:237)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:496)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:350)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:116)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
	at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:149)
	at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:237)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:496)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:350)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:116)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)
	at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
	at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:149)
	at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:237)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:496)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:350)
	at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:1980)
	at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1225)
	at com.googlecode.jmxtrans.util.JmxUtils.getJmxProcess(JmxUtils.java:494)
	at com.googlecode.jmxtrans.JmxTransformer.processFilesIntoServers(JmxTransformer.java:352)
	... 4 more

说是解析 com.googlecode.jmxtrans.util.LifecycleException 异常，这里需要我们自己编译一下jar包，在文章开头我们下载过jmxtrans源码

在  项目目录下进行编译，文末有我编译好的jar包

mvn clean package -Dmaven.test.skip=true -DskipTests=true;

编译完成我们需要的jar包在 jmxtrans-jmxtrans-parent-271\jmxtrans\target 目录下

jmxtrans-271-all.jar 就是我们需要用的jar包

将jar包传到jmxtrans目录下

我们对比一下发现我们编译的包是有这个类的，而他自带的那个没有

[root@ecs-t-001-0001 jmxtrans]# grep 'com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory' ./jmxtrans-271-all.jar 
Binary file ./jmxtrans-271-all.jar matches
[root@ecs-t-001-0001 jmxtrans]# grep 'com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory' ./jmxtrans-all.jar 
[root@ecs-t-001-0001 jmxtrans]#

替换 jmxtrans.sh 中应用的 jmxtrans jar包名称

#JAR_FILE=${JAR_FILE:-"jmxtrans-all.jar"}
JAR_FILE=${JAR_FILE:-"jmxtrans-271-all.jar"}

再次启动即可

 

验证jmxtrans是否成功运行

进入 influx jmxDB数据库(我们之前创建的)，查看表 show MEASUREMENTS，我们发现自动创了jvmMemory

[root@ecs-t-001-0001 jmxtrans]# influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
> show databases
name: databases
name
----
_internal
jmxDB
> use jmxDB
Using database jmxDB
> show 
ERR: error parsing query: found EOF, expected CONTINUOUS, DATABASES, DIAGNOSTICS, FIELD, GRANTS, MEASUREMENT, MEASUREMENTS, QUERIES, RETENTION, SERIES, SHARD, SHARDS, STATS, SUBSCRIPTIONS, TAG, USERS at line 1, char 6
> show MEASUREMENTS
name: measurements
name
----
jvmMemory

 具体查看数据，发现有数据写入

至此 jmxtrans 已成功监控 Kafka JMX端口，离成功更近了

 

安装 Grafana

 安装

yum install grafana-6.7.3-1.x86_64.rpm

配置文件默认路径 /etc/grafana/grafana.ini

 修改下web端口

> vim /etc/grafana/grafana.ini


# web页面端口默认3000
http_port = 9099

 启动服务并设置为开机启动

systemctl start grafana-server
systemctl enable grafana-server

查看启动状态

systemctl  status grafana-server

访问 web页面 ，第一次登陆账号密码 是 admin/admin ，登陆完成后会提示你设置密码

开始配置Grafana显示模板

点击DataSource

选择添加 InfluxDB ，并填写基本信息

 填写 influxDB 数据库信息

 点击保存

此时InfluxDB配置完成 ，我们几区创建仪表盘

选择 添加一个查询

 

选择我们上面设置的KafkaMonitor（因为我们只有一个默认的也是这个）

 

 

简单修改一下sql，因为我们上面只监控了JMX中的内存信息

注意：如果上面使用tag进行划分，这里就不要设置了，否则都叫Jvm内存使用了

 

简单配置完成，我们对比一下监控与jconsole监控对比

 

 

influxDB 与 Grafana 监控模板

 

模板下载

链接：https://pan.baidu.com/s/1ld-Yhv7wVutRxslV084GoQ 
提取码：0pzr 
复制这段内容后打开百度网盘手机App，操作更方便哦