Logstash分析App日志

五、Logstash分析App日志

5.1 概述

APP日志，主要是用来记录用户的操作，大体内容如下:
[INFO] 2019-12-28 04:53:36 [cn.oldxu.dashboard.Main] - DAU|8329|领取优惠券|2019-12-28 03:18:31

5.2 APP日志收集架构

5.3 步骤

1.首先通过 Filebeat 读取日志文件中的内容，并且将内容发送给 Logstash；
2.Logstash接收到内容后，将数据转换为结构化数据。然后输出给Elasticsearch；
3.Kibana添加Elasticsearch索引，读取数据，然后在Kibana中进行分析，最后进行展示；

5.4 APP日志收集实践

1.启动app产生日志
下载日志文件
wget http://cdn.xuliangwei.com/app-2020-08.log 

2.配置filebeat
cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths: /var/log/app.log

output.logstash:
  hosts: ["172.16.1.151:5044"]

3.配置logstash
cat app_logstash.conf 
input {
	beats {
	 port => 5044
	}
}

filter {
	mutate {
		split => { "message" => "|" }
		add_field => {

			"UserID" => "%{[message][1]}"
			"Action" => "%{[message][2]}"
			"Date" => "%{[message][3]}"
	}
	convert => {
			"UserID" => "integer"
			"Action" => "string"
			"Date" => "string"	
		}
	remove_field => ["message"]
	}
	date {
		#2020-08-28 01:05:02
		match => ["Date", "yyyy-MM-dd HH:mm:ss"]
		target => "@timestamp"
		timezone => "Asia/Shanghai"
	}
}
output {
	stdout {
		codec => rubydebug
	}
	elasticsearch {
		hosts => ["172.16.1.162:9200","172.16.1.163:9200","172.16.1.161:9200"]
		index => "app-%{+YYYY.MM.dd}"
		template_overwrite => true
	}
}

4.kibana上创建app索引展示效果