Elastic+logstash+filebeat做Nginx日志分析

一、Elasticserach安装 1、Installation(elastic 6.3.2 版本 依赖 java JDK8)

下载合适的版本： curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz tar -xvf elasticsearch-6.4.0.tar.gz cd elasticsearch-6.4.0/bin ./elasticsearch

2、集群健康检查 curl -XGET http://127.0.0.1:9200/_cat/health?v 集群健康查看 curl -XGET http://127.0.0.1:9200/_cat/nodes?v 节点状态查看 curl -XGET http://127.0.0.1:9200/_cat/indices?v 查看索引 curl -XPUT http://127.0.0.1:9200/customer?pretty 添加customer索引 curl -XDELETE http://127.0.0.1:9200/customer?pretty 删除索引

3、Config cluster.name: 集群名 node.name: 节点名 path.data: 数据存储路径 path.logs: 日志存储路径 network.host: 监听地址 http.port: 监听端口 JVM 配置、logging配置 参考官方文档 重要参数：

Path settings Cluster name Node name Network host Discovery settings Heap size Heap dump path GC logging Temp directory 重要的系统参数：

Disable swapping Increase file descriptors Ensure sufficient virtual memory Ensure sufficient threads JVM DNS cache settings

4、Running as daemon ./bin/elasticsearch -d -p pid 启动 kill cat pid 停止

5、Set up X-Pack 参考x-pack破解方法

二、kibana安装及配置

		wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz

tar -xzf kibana-6.4.0-linux-x86_64.tar.gz cd kibana-6.4.0-linux-x86_64/

		1、启动  ./bin/kibana   

		2、config：

server.port: 5601 server.host: "192.168.12.81" kibana.index: ".kibana" elasticsearch.username: "elastic" elasticsearch.password: "dinpay"

三、logstash安装及配置 wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz

配置 filebeat 收集Nginx日志： （Nginx.conf）

input { beats { port => 5044 codec => "json" } }

output { elasticsearch { hosts => ["127.0.0.1:9200"] index => "test1-nginx-access-%{+YYYY.MM.dd}" template_overwrite => true user => elastic password => dinpay } }

配置logstash.yml elastic安装在本机（没有配置https）

xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: elastic xpack.monitoring.elasticsearch.password: dinpay xpack.monitoring.elasticsearch.url: ["http://127.0.0.1:9200"]

四、filebeat安装及配置

wget https://artifacts.elastic.co/downloads/beats/filebeat/https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz

tar -xf filebeat-6.4.0-linux-x86_64.tar.gz

cd filebeat-6.4.0-linux-x86_64

配置 filebeat.yml

• type: log paths:

  • /var/log/nginx/access.log

output.logstash:

hosts: ["192.168.12.81:5044"]

xpack.monitoring: enabled: true elasticsearch: hosts: ["http://192.168.12.81:9200", "http://192.168.12.81:9200"] username: elastic password: dinpay

五、Nginx 日志格式配置：

log_format json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$clientRealIp",' '"remote_user":"$remote_user",' '"request":"$request",' '"http_user_agent":"$http_user_agent",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"url":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"status":"$status",' '"ss":"$upstream_status"}';

access_log  /var/log/nginx/access.log  json;

map $http_x_forwarded_for $clientRealIp {
	
""  $remote_addr;

~^(?P<firstAddr>[0-9\.]+),?.*$    $firstAddr;

}