elk收集 nginx集群日志 elk收集tomcat日志信息

elk收集tomcat和docker的日志

• 1. 收集tomcat的日志

• 1.1 tomcat的安装
• 1.2 日志的收集

• 2. 收集java的日志
• 3. docker日志的收集

• 3.1 docker的安装
• 3.2 单个容器的收集
• 3.3 多个容器的收集
• 3.4 容器的错误日志与正确日志的区分

1. 收集tomcat的日志

1.1 tomcat的安装

yum的安装方式

yum install tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp tomcat-javadoc -y

其他的安装方式

启动

systemctl start tomcat
systemctl status tomcat

检查端口
lsof -i:8080

访问测试

192.168.80.40:8080

1.2 日志的收集

我们收集的最好使用json的格式
我们修改tomcat的日志文件的格式

vim /etc/tomcat/server.xml
cat -n /etc/tomcat/server.xml
----------------
   137          <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
   138                 prefix="localhost_access_log." suffix=".txt"
   139              
      pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>

清空日志文件重启

>  /var/log/tomcat/localhost_access_log.2021-11-19.txt
 systemctl restart tomcat

访问，点击才会出现日志
192.168.80.40:8080

查看日志

[root@yw7 opt]# tail -f /var/log/tomcat/localhost_access_log.2021-11-19.txt 


{"clientip":"192.168.80.1","ClientUser":"-","authenticated":"-","AccessTime":"[19/Nov/2021:03:21:32 -0500]","method":"GET /docs/security-howto.html HTTP/1.1","status":"200","SendBytes":"43375","Query?string":"","partner":"http://192.168.80.40:8080/","AgentVersion":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"}

json格式检验
{
	"clientip": "192.168.80.1",
	"ClientUser": "-",
	"authenticated": "-",
	"AccessTime": "[19/Nov/2021:03:21:32 -0500]",
	"method": "GET /docs/security-howto.html HTTP/1.1",
	"status": "200",
	"SendBytes": "43375",
	"Query?string": "",
	"partner": "http://192.168.80.40:8080/",
	"AgentVersion": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
}

filebeat的文件配置

cd /etc/filebeat 
vim filebeat.yml

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    -  /var/log/tomcat/localhost_access_log*
  json.keys_under_root: true
  json.overwrite_keys: true
setup.kibana:
  host: "192.168.80.40:5601"
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "tomcat-%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "tomcat"
setup.template.pattern: "tomcat-*"
setup.template.enabled: false
setup.template.overwrite: true

启动es 和filebeat

systemctl start elasticsearch 
systemctl start filebeat

kibana查看

systemctl start kibana

http://192.168.80.40:5601/

信息的筛选

2. 收集java的日志

这里我们用es的日志做实验
https://www.elastic.co/guide/en/beats/filebeat/6.6/multiline-examples.html

filebeat的配置文件

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/elasticsearch/elasticsearch.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
setup.kibana:
  host: "192.168.80.40:5601"
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "es-%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true

解释

multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

多行模式：就是不规则日志文件时怎么进行分别，一行还是多行

对于json的模式和多行模式都是加到一个的type的下边，不知全局都有作用（这点需要注意）

systemctl restart filebeat

kibana

3. docker日志的收集

3.1 docker的安装

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#g' /etc/yum.repos.d/docker-ce.repo

yum install docker-ce -y
systemctl start docker

vi /etc/docker/daemon.json
{
  "registry-mirrors": ["https://registry.docker-cn.com"]
  }
systemctl restart docker

运行docker

docker pull nginx
docker run --name nginx -p 80:80 -d nginx
docker ps
docker logs -f nginx

3.2 单个容器的收集

filebeat的配置文件
https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-input-docker.html

filebeat.inputs:
- type: docker
  containers.ids: 
    - 'bf839addeff3780c6908a8062c46887a17d81a3942344792c3b1cda2bc3acd29'
  tags: ["docker-nginx"] 
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

systemctl restart filebeat

messages是在一起的，不知json格式的

正确错误日志在一起等等问题

"stream": "stderr" 错误
"stream": "stdout" 正确的

3.3 多个容器的收集

docker commit nginx nginx:v2
docker images
docker run --name nginx1 -p 8888:80 -d nginx:v2

容器目录下边有两个目录

ls /var/lib/docker/containers/
[root@yw7 filebeat]#  ls /var/lib/docker/containers/
66000b751bab28a98841475621c5a9ffdfce4b0032a4e8fb2256f7211ab9841b
bf839addeff3780c6908a8062c46887a17d81a3942344792c3b1cda2bc3acd29

filebeat配置文件

如果直接配置filebeat存到es里本台机器所有的容器日志都会混在一起没有办法区分
多容器日志收集处理：
其实收集的日志本质来说还是文件，而这个日志是以容器-json.log命名存放在默认目录下的json格式的文件

[root@yw7 bf839addeff3780c6908a8062c46887a17d81a3942344792c3b1cda2bc3acd29]# ll
total 44
-rw-r----- 1 root root 16616 Nov 19 04:06 bf839addeff3780c6908a8062c46887a17d81a3942344792c3b1cda2bc3acd29-json.log
drwx------ 2 root root     6 Nov 19 03:59 checkpoints
-rw------- 1 root root  2902 Nov 19 04:11 config.v2.json
-rw-r--r-- 1 root root  1512 Nov 19 04:11 hostconfig.json
-rw-r--r-- 1 root root    13 Nov 19 03:59 hostname
-rw-r--r-- 1 root root   174 Nov 19 03:59 hosts
drwx--x--- 2 root root     6 Nov 19 03:59 mounts
-rw-r--r-- 1 root root    57 Nov 19 03:59 resolv.conf
-rw-r--r-- 1 root root    71 Nov 19 03:59 resolv.conf.hash

但是每个容器的ID都不一样，为了区分不同服务运行的不同容器，
可以使用docker-compose通过给容器添加labels标签来作为区分
然后filbeat把容器日志当作普通的json格式来解析并传输到es

安装：

国内主要镜像地址如下：

清华：https://pypi.tuna.tsinghua.edu.cn/simple
阿里云：https://mirrors.aliyun.com/pypi/simple/
中国科技大学 https://pypi.mirrors.ustc.edu.cn/simple/
华中理工大学：http://pypi.hustunique.com/
山东理工大学：http://pypi.sdutlinux.org/
豆瓣：http://pypi.douban.com/simple/

yum install -y python2-pip
pip install -i hhttps://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install docker-compose
docker-compose version

编写docker-compose.yml

cat docker-compose.yml 
version: '3'
services:
  nginx:
    image: nginx:v2
    # 设置labels
    labels:
      service: nginx
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
  db:
    image: nginx:latest
    # 设置labels
    labels:
      service: db 
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"



version: '3'
services:
  nginx:
    image: nginx:v2
    labels:
      service: nginx
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
  db:
    image: nginx:latest
    labels:
      service: db 
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"

执行，必须在yml文件的路径下

docker-compose up -d

查看

docker ps

filebeat的配置文件

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
output.elasticsearch:
  hosts: ["192.168.47.175:9200"]
  indices:
    - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        attrs.service: "nginx"
    - index: "docker-db-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        attrs.service: "db"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

为什么是 when.contains:
attrs.service: “db”

/var/lib/docker/containers/3e93185d03baf52453c5005f0dd9a84a25f8ae0fccf4b0a0e953cbf814025545
cat 3e93185d03baf52453c5005f0dd9a84a25f8ae0fccf4b0a0e953cbf814025545-json.log

{"log":"192.168.80.1 - - [19/Nov/2021:11:45:28 +0000] \"GET /yb.html HTTP/1.1\" 404 555 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36\" \"-\"\n","stream":"stdout","attrs":{"service":"db"},"time":"2021-11-19T11:45:28.531578546Z"}

json解析
{
	"log": "192.168.80.1 - - [19/Nov/2021:11:45:28 +0000] \"GET /yb.html HTTP/1.1\" 404 555 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36\" \"-\"\n",
	"stream": "stdout",
	"attrs": {
		"service": "db"
	},
	"time": "2021-11-19T11:45:28.531578546Z"
}

因为service在sttrs的里边

3.4 容器的错误日志与正确日志的区分

通过不同的定义标签类区别不同的docker容器
还有一个问题，正确和错误的日志还是在一起
观察正确日志与错误的区别

"stream": "stderr" 错误
"stream": "stdout" 正确的

配置filebeat的配置文件

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
output.elasticsearch:
  hosts: ["192.168.80.40:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "nginx"
          stream: "stdout"
    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "nginx"
          stream: "stderr"
    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "db"
          stream: "stdout"
    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "db"
          stream: "stderr"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

systemctl restart filebeat