一、安装环境说明

环境阿里云ECS centos7.6 x86_64 最小化安装

涉及到的服务端口提前在安全组内放开允许内网访问

三台服务器服务部署情况:

tidb04  192.10.0.59 安装 prometheus,grafana,node_exporter,mysqld_exporter
tidb05 192.10.0.246 安装 node_exporter,mysqld_exporter
tidb06 192.10.0.247 安装 node_exporter
特别提示:三台服务器主机名都绑定本地hosts文件

二、prometheus安装

2.1 安装包下载

mkdir -p /data/soft/prometheus/data
cd /data/soft/prometheus/
wget https://github-releases.githubusercontent.com/6838921/4063cb00-da9b-11eb-8735-e43628dda2bc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210712%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210712T041623Z&X-Amz-Expires=300&X-Amz-Signature=47d660187c9a5410afb6acb2ae3da60ce26fcc42a63cb36a94f02d388ccf2266&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.28.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
tar -xvf prometheus-2.28.1.linux-amd64.tar.gz
mv prometheus-2.28.1.linux-amd64 prometheus
cd prometheus; mkdir data;

2.2 启动脚本

cat </etc/systemd/system/prometheus.service
[Unit]
Description="prometheus"
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
ExecStart=/data/soft/prometheus/prometheus --config.file=/data/soft/prometheus/prometheus.yml --storage.tsdb.path=/data/soft/prometheus/data --web.enable-lifecycle --enable-feature=remote-write-receiver --query.lookback-delta=2m

Restart=on-failure
RestartSecs=5s
SuccessExitStatus=0
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=prometheus

[Install]
WantedBy=multi-user.target
EOF

2.3 服务器启动命令

服务加载
systemctl daemon-reload
systemctl enable prometheus
启动服务
systemctl restart prometheus
服务状态查看
systemctl status prometheus

2.4 测试服务是否启动

浏览器访问 http://192.10.0.59:9090/targets

截图如下:代表启动成功

监控之Prometheus+grafana+node_exporter配置_linux


三、node_exporter安装

node_exporter 是prometheus的系统资源监控组件

3.1 node_exporter服务启动脚本

cat <<EOF >/etc/systemd/system/node_exporter.service

[Unit]
Description="node_exporter"
Documentation=https://github.com/prometheus/node_exporter
After=network.target

[Service]
WorkingDirectory=/opt/server/exporter
Type=simple
ExecStart=/opt/server/exporter/node_exporter/node_exporter --web.listen-address=:19100
Restart=on-failure
RestartSec=10s
SuccessExitStatus=0
LimitNOFILE=300000

[Install]
WantedBy=multi-user.target
EOF

3.2 服务器启动命令

服务加载
systemctl daemon-reload
systemctl enable node_exporter
启动服务
systemctl restart node_exporter
服务状态查看
systemctl status node_exporter

四、node_exporter 接入Prometheus

在scrape_configs块下添加如下内容:(注意参数格式缩进,否则导致服务启动失败)

scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['192.10.0.59:9090']
- job_name: 'node_exporter'
static_configs:
- targets: ['192.10.0.59:19100','192.10.0.246:19100','192.10.0.247:19100']

完整配置文件如下:

[root@tidb04 ~]# cat /data/soft/prometheus/prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['192.10.0.59:9090']
- job_name: 'node_exporter'
static_configs:
- targets: ['192.10.0.59:19100','192.10.0.246:19100','192.10.0.247:19100']

重启服务:

systemctl restart prometheus

验证node_exporter是否接入到prometheus:

​http://192.10.0.59:19100/metrics​

监控之Prometheus+grafana+node_exporter配置_服务器_02

添加的node_exporter的服务器都可以查到

五、Grafana安装

官网地址:

​https://grafana.com/grafana/download?edition=oss ​

Red Hat, CentOS, RHEL, and Fedora(64 Bit)系统采用下面的2种方式安装:

rpm包安装:
wget https://dl.grafana.com/oss/release/grafana-8.3.3-1.x86_64.rpm
sudo yum install grafana-8.3.3-1.x86_64.rpm

下载二进制包安装:
wget https://dl.grafana.com/oss/release/grafana-8.3.3.linux-amd64.tar.gz
tar -zxvf grafana-8.3.3.linux-amd64.tar.gz

本次演示安装采用的是rpm包安装

配置文件:可以指定管理账户名称和pass,可以接入ldap账户

[root@tidb04 soft]# egrep "ldap|admin_user|admin_password" /etc/grafana/grafana.ini  
;admin_user = admin

;admin_password = admin

[auth.ldap]

;config_file = /etc/grafana/ldap.toml

启动命令:

systemctl status grafana-server
systemctl stop grafana-server
/etc/init.d/grafana-server status/stop/start/restart

grafana接入prometheus数据源:

登录http://192.10.0.59:3000/login

监控之Prometheus+grafana+node_exporter配置_服务器_03

输入URL,点击save&test即可

六、导入node_exporter dashboard模版

登录 grafana官网查找node_exporter  dashboard的模板 导入到自建的grafana系统  

dashboard的模板查找地址:

https://grafana.com/grafana/dashboards

具体导入node_exporter dashboard模版配置过程可以参考:https://www.cnblogs.com/Pycainiao/p/14568910.html

监控之Prometheus+grafana+node_exporter配置_服务器_04

点击选中的中文版node_exporter dashboard 进入到详情中可以看到这个模版到ID为8919

监控之Prometheus+grafana+node_exporter配置_服务器_05

登录grafana控制台 选择左侧+号 选择下拉框Import:

监控之Prometheus+grafana+node_exporter配置_服务器_06

监控之Prometheus+grafana+node_exporter配置_linux_07

过几秒中,发现dashboard出现图,代表添加成功:

监控之Prometheus+grafana+node_exporter配置_服务器_08

七、删除已经存在的dashboard模板

点击左上角General/Home  进入搜索dashboard页面:

监控之Prometheus+grafana+node_exporter配置_服务器_09

监控之Prometheus+grafana+node_exporter配置_linux_10

监控之Prometheus+grafana+node_exporter配置_linux_11

监控之Prometheus+grafana+node_exporter配置_服务器_12