kafka_exporter 通过 Kafka Protocol Specification 收集 Brokers, Topics 以及 Consumer Groups的相关指标,使用简单,运行高效,相比于以往通过kafka内置的脚本进行收集,由于没有了JVM的运行开销,指标收集时间从分钟级别降到秒级别,便于大规模集群的监控。


git项目地址:https://github.com/danielqsj/kafka_exporter

下载地址: https://github.com/danielqsj/kafka_exporter/releases/download/v1.2.0/kafka_exporter-1.2.0.linux-amd64.tar.gz

docker安装略过

grafana安装

#docker run -d --name=grafana -v /etc/localtime:/etc/localtime:ro --restart=always -p 3000:3000 grafana/grafana 

prometheus启动

#docker run -d --name=prometheus -p 9090:9090 --restart=always -v /etc/localtime:/etc/localtime:ro  -v /home/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml  prom/prometheus

注:提前将需要挂载的目录创建好

#mkdir -pv /home/prometheus/
#touch /home/prometheus/prometheus.yml

登陆到kafka服务器下载kafka_exporter

#wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.2.0/kafka_exporter-1.2.0.linux-amd64.tar.gz

# tar -zxvf kafka_exporter-1.2.0.linux-amd64.tar.gz 

# cd kafka_exporter-1.2.0.linux-amd64

# ./kafka_exporter --kafka.server=kafkaIP或者域名:9092 & 

#ss -tunl

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Grafana

注:9308是kafka_exporter的端口

登陆到prometheus服务器

编写/home/prometheus/prometheus.yml文件

#vim /home/prometheus/prometheus.yml

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

   - job_name: 'prometheus'

     static_configs:

      - targets: ['localhost:9090']

 

  - job_name: 'kafka'

     static_configs:

      - targets: ['kafkaIP或者域名:9308']

        labels:

          instance: kafka@kafkaIP或者域名

重启prometheus

#docker restart prometheus

通过浏览器访问:http://prometheus服务器IP:9090,所添加的kafka_exporter状态为UP,就可以去配置grafana

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Grafana _02

配置grafana

通过浏览器访问:http://grafana服务器IP:3000,

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Prometheus_03

添加数据源,选择prometheus,填入prometheus服务器IP端口,点击保存

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Grafana _04

导入监控图表

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Prometheus_05

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Grafana _06

输入7589,光标往下移,如下图

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Prometheus_07

图表数据就出来了

Prometheus+Grafana+kafka_exporter搭建监控系统监控kafka_Grafana _08