Prometheus 学习笔记 - 监控 Application 案例配置

Prometheus 除了监控服务器节点，还可以对application和容器进行监控。下面看看4个例子，监控apache，监控 docker daemon，监控docker container和 监控 kubenetes

例1 监控Apache 相关服务

和监控Linux 节点类似，我们也需要安装对应的exporter
登陆进apache 服务器，创建新用户，下载exporter，解压，拷贝到可执行目录，更改权限

sudo useradd -M -r -s /bin/false apache_exporter
wget https://github.com/Lusitaniae/apache_exporter/releases/download/v0.7.0/apache_exporter-0.7.0.linux-amd64.tar.gz
tar xvfz apache_exporter-0.7.0.linux-amd64.tar.gz
sudo cp apache_exporter-0.7.0.linux-amd64/apache_exporter /usr/local/bin/
sudo chown apache_exporter:apache_exporter /usr/local/bin/apache_exporter

创建一个systemd的守护进程

sudo vi /etc/systemd/system/apache_exporter.service
[Unit]
Description=Prometheus Apache Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=apache_exporter
Group=apache_exporter
Type=simple
ExecStart=/usr/local/bin/apache_exporter

[Install]
WantedBy=multi-user.target

启动进程

sudo systemctl enable apache_exporter
sudo systemctl start apache_exporter
sudo systemctl status apache_exporter

手动查看一下是否成功，他应该会返回大量数据

curl localhost:9117/metrics

接下来，我们登陆prometheus服务器

sudo vi /etc/prometheus/prometheus.yml

添加下面的job

- job_name: 'Apache Server'
  static_configs:
  - targets: ['limedrop-apache:9117']

重启prometheus服务

sudo systemctl restart prometheus

访问网页即可看见结果

例2 监控 Docker Daemon

Docker Daemon 的监控就更容易了。我们只需要打开一个配置文件的设置就行了

sudo vi /etc/docker/daemon.json

添加下面的内容

{
  "experimental": true,
  "metrics-addr": "10.0.1.102:9323"
}

重启

sudo systemctl restart docker

查看一下

curl 10.0.1.102:9323/metrics

回到 prometheus服务器修改一下配置文件

sudo vi /etc/prometheus/prometheus.yml

添加内容

- job_name: 'Docker'
  static_configs:
  - targets: ['limedrop-docker:9323']

重启

sudo systemctl restart prometheus

访问结果，显示docker的运行状态

例3 监控 docker Container

我们可以用一个叫做 cAdvisor的服务来监控状态
直接在host上安装启动这个容器

docker run -d --restart always --name cadvisor -p 8080:8080 -v "/:/rootfs:ro" -v "/var/run:/var/run:rw" -v "/sys:/sys:ro" -v "/var/lib/docker/:/var/lib/docker:ro" google/cadvisor:latest

确认一下

curl localhost:8080/metrics

查看一下当前的容器

cloud_user@ip-10-0-1-102:~$ docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                    NAMES
5a91ebfef3d2        google/cadvisor:latest   "/usr/bin/cadvisor -…"   7 minutes ago       Up 7 minutes        0.0.0.0:8080->8080/tcp   cadvisor
329c59ddfdee        nginx                    "/docker-entrypoint.…"   13 minutes ago      Up 13 minutes       80/tcp                   web2
99926c15aa01        nginx                    "/docker-entrypoint.…"   13 minutes ago      Up 13 minutes       80/tcp                   web1

切回我们的prometheus 服务器，修改配置

sudo vi /etc/prometheus/prometheus.yml

添加内容

- job_name: 'Docker Containers'
  static_configs:
  - targets: ['limedrop-docker:8080']

重启服务

sudo systemctl restart prometheus

查看一下我们的web容器的内存

container_memory_usage_bytes{name=~"web."}

例4 监控 kubernetes

我们使用kube-state-metrics来进行监控，首先从github下载代码，进入文件目录，然后checkout到对应的版本号

git clone https://github.com/kubernetes/kube-state-metrics.git
cd kube-state-metrics/
git checkout v1.8.0

执行kubectl 生成对应的服务

kubectl apply -f kubernetes

接下来需要自定义一个nodeport的服务

cd ..
vi kube-state-metrics-nodeport-svc.yml

yaml文件如下

kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: kube-state-nodeport
spec:
  selector:
    k8s-app: kube-state-metrics
  ports:
  - protocol: TCP
    port: 8080
    nodePort: 30000
  type: NodePort

执行

kubectl apply -f kube-state-metrics-nodeport-svc.yml

测试

curl localhost:30000/metrics

切换回Prometheus服务器，修改配置

sudo vi /etc/prometheus/prometheus.yml

添加内容

- job_name: 'Kubernetes'
  static_configs:
  - targets: ['limedrop-kube:30000']

重启服务

sudo systemctl restart prometheus

访问查看一下我们的pod状态

kube_pod_status_ready{namespace="default",condition="true"}