Prometheus+Grafana+cAdvisor实现监控Pod

K爷 DevOps视角

  • 安装Prometheus
  • 部署node_exporter
  • 配置Prometheus采集master和node节点数据
  • 部署cadvisor
  • 准备镜像
  • 容器方式部署
  • 配置Prometheus搜集cadvisor数据
  • Grafana添加监控

监控pod资源

Prometheus与Grafana部署按照官方提供的部署方式部署即可。 cadvisor是一个谷歌开源的容器监控工具，cadvisor不仅可以搜集一台机器上所有运行的容器信息，还提供基础查询界面和http接口， 方便其他组件如Prometheus进行数据抓取。

安装Prometheus


# cd /usr/local/src/
# wget https://github.com/prometheus/prometheus/releases/download/v2.17.1/prometheus-2.17.1.linux-amd64.tar.gz
# tar xf prometheus-2.17.1.linux-amd64.tar.gz 
# ln -sv /usr/local/src/prometheus-2.17.1.linux-amd64 /usr/local/src/prometheus
# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server 
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/usr/local/src/prometheus/
ExecStart=/usr/local/src/prometheus/prometheus --config.file=/usr/local/src/prometheus/prometheus.yml

[Install] 
WantedBy=multi-user.target
# systemctl start prometheus
# systemctl enable prometheus

部署node_exporter

在master节点和node节点部署


# cd /usr/local/src/

# wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
# tar xf node_exporter-0.18.1.linux-amd64.tar.gz 
# ln -s /usr/local/src/node_exporter-0.18.1.linux-amd64 /usr/local/src/node_exporter
# vim /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/usr/local/src/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target
# systemctl start node-exporter.service
# systemctl enable node-exporter

配置Prometheus采集master和node节点数据



# cd /usr/local/src/prometheus
# vim prometheus.yml
scrape_configs:
......
  - job_name: 'k8s-node'
    static_configs:
    - targets: ['172.16.1.33:9100', '172.16.1.34:9100', '172.16.1.35:9100']
    
  - job_name: 'k8s-master'
    static_configs:
    - targets: ['172.16.1.30:9100', '172.16.1.31:9100', '172.16.1.32:9100']
# systemctl restart prometheus

部署cadvisor

准备镜像


# docker pull google/cadvisor:v0.33.0
# docker tag google/cadvisor:v0.33.0 harbor.kevin.com/base/cadvisor:v0.33.0
# docker push harbor.kevin.com/base/cadvisor:v0.33.0

容器方式部署

在各node节点运行


# docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
harbor.kevin.com/base/cadvisor:v0.33.0

在浏览器输入http://:8080即可访问

配置Prometheus搜集cadvisor数据


# vim /usr/local/src/prometheus/prometheus.yml 
......
  - job_name: 'k8s-pods'
    static_configs:
    - targets: ['172.16.1.33:8080', '172.16.1.34:8080', '172.16.1.35:8080']
# systemctl restart prometheus

Grafana添加监控