部署规划

主机名

bigdata111

bigdata112

bigdata113

服务部署


Kafka、

Prometheus、

Grafana、

Kafka exporter


Kafka

Kafka


Kafka

部署目录


/opt/kafka_2.11-2.1.1


Prometheus

部署目录


/opt/prometheus/


Kafka exporter

部署目录


/opt/kafka_exporter-1.4.2.linux-amd64

Grafana安装参见

​Grafana8.5.0+Zabbix5.0LTS可视化监控数据_leboop-L的博客-CSDN博客https://dl.grafana.com/enterprise/release/grafana-enterprise-8.5.0-1.x86_64.rpmhttps://dl.grafana.com/enterprise/release/grafana-enterprise-8.5.0-1.x86_64.rpmIndex of /grafana/ | 清华大学开源软件镜像站 | Tsinghua Open Source Mirrorjavascript:void(0)​​Prometheus安装参见

​Centos7安装Prometheus_leboop-L的博客-CSDN博客目录Prometheus整体架构下载安装配置启动验证报错解决Prometheus整体架构下载 官方下载地址如下:Download | Prometheushttps://prometheus.io/download/下载版本prometheus-2.35.0-rc0.linux-amd64.tar.gz,如图:安装 将下载包prometheus-2.35.0-rc0.linux-amd64.tar.gz上传到/...javascript:void(0)​

安装Kafka exporter

下载

        关于exporter介绍参见

​Exporters and integrations | PrometheusAn open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.https://prometheus.io/docs/instrumenting/exporters/​​本文使用kafka exporter版本如下:

kafka_exporter-1.4.2.linux-amd64.tar.gz

下载地址:

​Releases · danielqsj/kafka_exporter · GitHubPrometheus+Grafana监控Kafka_linuxhttps://github.com/danielqsj/kafka_exporter/releases​

安装

        将下载的包上传到bigdata111的/root/toos目录下,执行如下命令解压:

tar -zxvf kafka_exporter-1.4.2.linux-amd64.tar.gz -C /opt/

Prometheus+Grafana监控Kafka_kafka_02

Prometheus+Grafana监控Kafka_linux_03 

启动

        执行如下命令,启动kafka_exporter:

./kafka_exporter --kafka.server bigdata112:9092

Prometheus+Grafana监控Kafka_prometheus_04 kafka_exporter监听端口9308。

浏览器中输入​​http://bigdata111:9308/metrics​​,可以看到kafka_exporter采集的监控指标,如图:

Prometheus+Grafana监控Kafka_prometheus_05


 prometheus配置

配置

        打开prometheus配置文件,命令如下:

vi /opt/prometheus/prometheus.yml

配置文件添加如下配置:  

- job_name: 'kafka'
static_configs:
- targets: ['bigdata111:9308'] #配置kafka监控

添加好后,如图: 

Prometheus+Grafana监控Kafka_prometheus_06

执行如下命令,检查配置格式是否正确: 

./promtool check config prometheus.yml

Prometheus+Grafana监控Kafka_kafka exporter_07

重启

        执行如下命令重启prometheus,并查看服务状态:

systemctl restart prometheus.service

systemctl status prometheus.service

如图:

Prometheus+Grafana监控Kafka_kafka_08


Prometheus查看Kafka指标

浏览器输入Prometheus服务地址

​http://bigdata111:9090/​

点击Status-->Targets,如图:

Prometheus+Grafana监控Kafka_prometheus_09

如图:

Prometheus+Grafana监控Kafka_linux_10

Grafana展示Kafka指标

Grafana添加Prometheus数据源

        Configuration-->Data sources-->Add data source,如图: 

Prometheus+Grafana监控Kafka_kafka exporter_11

 URL配置Prometheus的服务器地址和端口。点击Save & test,如图:

Prometheus+Grafana监控Kafka_linux_12


导入Dashboard

Kafka Exporter的Dashboard参见

​Kafka Exporter Overview dashboard for Grafana | Grafana LabsKafka resource usage and throughputPrometheus+Grafana监控Kafka_kafka exporter_13https://grafana.com/grafana/dashboards/7589​

在Grafana如下导入,点击Create-->Import

Prometheus+Grafana监控Kafka_linux_14

日安如7589,如图:

Prometheus+Grafana监控Kafka_prometheus_15

 点击右边Load,如图:

Prometheus+Grafana监控Kafka_linux_16

 点击Import, 如图:

Prometheus+Grafana监控Kafka_prometheus_17

需要配置kafka exporter地址端口,如图:

Prometheus+Grafana监控Kafka_kafka_18

 数据成功显示,如图:

Prometheus+Grafana监控Kafka_kafka exporter_19