Prometheus + node_exporter +process_exporter + CM Datasource + Grafana 监控安装部署
一、node_exporter安装与配置
node_exporter是Prometheus的一个插件，部署该插件，可以将本地机器的系统资源信息筹集并发送给Prometheus。
在需要监控的机器上面都要安装该插件。
这里我只演示监控我的两台机器
192.168.0.113,
192.168.0.114
进入到http://prometheus.io/download官网,或者使用我给的安装包
#下载安装包并上传服务器/software：node_exporter-1.3.0.linux-amd64.tar.gz
1.1.解压
cd /software
tar -zxvf node_exporter-1.3.0.linux-amd64.tar.gz -C /opt/
cd /opt/
mv node_exporter-1.3.0.linux-amd64 node_exporter
cd node_exporter/
#启动node_exporter
# nohup ./node_exporter --web.listen-address=":9100" >/dev/null 2>&1 &
#注意：--web.listen-address=":9100" 端口一定不能被占用，否则会报错，可以自己手动修改为任意端口1.2.添加到系统服务中
vim /etc/systemd/system/node_exporter.service-------------------------------配置如下内容--------------
[Unit]
Description=Node Export
After=network.target
Documentation=https://prometheus.io/docs/guides/node-exporter/
[Service]
Type=simple
WorkingDirectory=/tmp/
ExecStart=/opt/node_exporter/node_exporter \
--web.listen-address=:9100
Restart=on-failure
[Install]
WantedBy=multi-user.target#注意：–web.listen-address=:9100 端口一定不能被占用，否则会报错，可以自己手动修改为任意端口
#授权可执行
chmod +x /etc/systemd/system/node_exporter.service
#重载systemd系统
systemctl daemon-reload
#启动服务
systemctl start node_exporter
systemctl status node_exporter
systemctl enable node_exporter
#检验是否安装成功，登录ip:9100进行查看二、processes_exporter安装与配置
process_exporter是Prometheus的一个插件，部署该插件，可以将本地机器的进程信息筹集并发送给Prometheus。
需要监控的机器上面都要安装该插件。
2.1.下载
下载地址：
https://github.com/ncabatoff/process-exporter/releases/tag/v0.5.0 下载：process-exporter-0.5.0.linux-amd64.tar.gz
2.2.上传服务器并解压
这里我已经提前上传到/software
cd /software
tar -zxvf process_exporter-0.5.0.linux-amd64.tar.gz -C /opt/
cd /opt/
mv process_exporter-0.5.0.linux-amd64.tar.gz process_exporter
cd process_exporter/2.3.创建process-name.yaml
cd /opt/process_exportervi process-name.yaml //将下面内容写入，也可自己再下面添加相关的进程
process_names:
- name: "{{.Matches}}"
cmdline:
- 'DataNode'
- name: "{{.Matches}}"
cmdline:
- 'kafka'
- name: "{{.Matches}}"
cmdline:
- 'SecondaryNameNode'
- name: "{{.Matches}}"
cmdline:
- 'NodeManager'
- name: "{{.Matches}}"
cmdline:
- 'greenplum'
- name: "{{.Matches}}"
cmdline:
- 'hadoop'
- name: "{{.Matches}}"
cmdline:
- 'hbase'
- name: "{{.Matches}}"
cmdline:
- 'hudi'
- name: "{{.Matches}}"
cmdline:
- 'hive'
- name: "{{.Matches}}"
cmdline:
- 'zookeeper'
- name: "{{.Matches}}"
cmdline:
- 'QueryServer'
- name: "{{.Matches}}"
cmdline:
- 'QuorumPeerMain'
- name: "{{.Matches}}"
cmdline:
- 'java'2.4.添加到系统服务中
vim /etc/systemd/system/process_exporter.service-------------------------------配置如下内容--------------
[Unit]
Description=process-exporter
[Service]
Type=simple
WorkingDirectory=/tmp/
ExecStart=/opt/process_exporter/process_exporter \
--config.path=/opt/process_exporter/process-name.yaml \
--web.listen-address=:9256
Restart=on-failure
[Install]
WantedBy=multi-user.target#注意：–web.listen-address=:9256 端口一定不能被占用，否则会报错，可以自己手动修改为任意端口
#授权可执行
chmod +x /etc/systemd/system/process_exporter.service
#重载systemd系统
systemctl daemon-reload
#启动服务
systemctl start process_exporter
systemctl status process_exporter
systemctl enable process_exporter#检验是否安装成功，登录ip:9256进行查看
**
三、Prometheus安装与配置
**
进入到http://prometheus.io官网
下载最新版本 prometheus-2.31.1.linux-amd64.tar.gz
这里只需要在一台机器上安装就好
//下载安装包：prometheus-2.31.1.linux-amd64.tar.gz
3.1.解压文件到/opt/目录下
tar -zxvf prometheus-2.31.1.linux-amd64.tar.gz -C /opt/3.2.重命名
cd /opt
mv prometheus-2.31.1.linux-amd64 prometheus
cd prometheus3.3.配置prometheus服务端
vim prometheus.yml# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration:告警配置，集成alertmanager插件
alerting:
alertmanagers:
- static_configs:
- targets:
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: "prometheus"
static_configs:
- targets: ["192.168.0.113:9090"] #为当前机器ip
#通过配置file 获取target
- job_name: 'node_exporter'
file_sd_configs:
- files:
- /opt/prometheus/targets/nodes/*.json3.4.创建我们的动态配置文件
vim /opt/prometheus/targets/nodes/host.json[
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "113node",
"group": "node_exporter",
"host_ip": "192.168.0.113",
"hostname": "node2"
},
"targets": [
"192.168.0.113:9101"
]
},
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "114node",
"group": "node_exporter",
"host_ip": "192.168.0.114",
"hostname": "node3"
},
"targets": [
"192.168.0.114:9110"
]
},
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "111node",
"group": "node_exporter",
"host_ip": "192.168.0.111",
"hostname": "master"
},
"targets": [
"192.168.0.111:9100"
]
}
]vim /opt/prometheus/targets/nodes/process.json[
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "113node",
"group": "process_exporter",
"host_ip": "192.168.0.113",
"hostname": "node2"
},
"targets": [
"192.168.0.113:9256"
]
},
{
"labels": {
"origin_prometheus": "prometheus",
"desc": "114node",
"group": "process_exporter",
"host_ip": "192.168.0.114",
"hostname": "node3"
},
"targets": [
"192.168.0.114:9256"
]
}
]3.5.添加peometheus到系统服务脚本中
vim /etc/systemd/system/prometheus.service[Unit]
Description=Promtheus
Documentation=https://prometheus.io/
After=network.target
[Service]
User=root
ExecStart=/opt/prometheus/prometheus \
--config.file=/opt/prometheus/prometheus.yml \
--web.listen-address=:9090
Type=simple
Restart=on-failure
[Install]
WantedBy=multi-user.target#启动脚本授权可执行
chmod +x /etc/systemd/system/prometheus.service
#系统服务配置重新加载
systemctl daemon-reload
#服务添加可随机启动
systemctl enable prometheus.service
#启动服务
systemctl start prometheus.service
#查看服务状态
systemctl status prometheus.service3.6.登录验证：192.168.0.113:9090 如下
四、Grafana安装与配置
4.1.安装grafana
#下载安装包：grafana-8.2.5-1.x86_64.rpm 并上传到服务器
#1.使用yum安装
yum localinstall grafana-8.2.5-1.x86_64.rpm#启动Grafana
systemctl start grafana-server
#停止Grafana
systemctl stop grafana-server
#重启Grafana
systemctl restart grafana-server
#设置开机自启动
systemctl enable grafana-server#登录：192.168.0.113:3000,首次登录账号密码为：admin/admin
4.2.安装CM Datasource
将cloudera manager的数据信心作为datasource接入grafana进行可视化监控。
4.2.1安装插件
方法一：github下载，解压到指定位置
https://grafana.com/grafana/plugins/foursquare-clouderamanager-datasource/installation4.2.2
下载完成后，上传服务器，解压到/var/lib/grafana/plugins
unzip foursquare-clouderamanager-datasource-0.9.3.zip方法二：直接命令行安装
grafana-cli plugins install foursquare-clouderamanager-datasource4.2.3重启grafana
systemctl restart grafana-server延伸：grafana插件
grafana-cli plugins install grafana-clock-panel #时钟
grafana-cli plugins install kentik-app #网络流量
grafana-cli plugins install grafana-piechart-panel #饼图
grafana-cli plugins install novalabs-annotations-panel #注释
grafana-cli plugins install farski-blendstat-panel #混合
grafana-cli plugins install yesoreyeram-boomtable-panel #多表
grafana-cli plugins install yesoreyeram-boomtheme-panel #多主题
grafana-cli plugins install jeanbaptistewatenberg-percent-panel #百分比
grafana-cli plugins install corpglory-progresslist-panel #进程列表
grafana-cli plugins install mxswat-separator-panel #分隔符
grafana-cli plugins install aidanmountford-html-panel #网页4.3.配置grafana
4.3.1添加数据源
配置prometheus数据源
配置CM插件数据源，前提：我们的cdh已经部署完成
4.3.2制作监控看板
这里我们直接将我给的模板导入即可。
4.4.看板展示
最终的页面如图展示
