ELK
一、ELK介绍
需求背景：随着业务发展越来越大，服务器会越来越多，那么，各种日志量（比如，访问日志、应用日志、错误日志等）会越来越多。 因此，开发人员排查问题，需要到服务器上查看日志，很不方便。而运维人员也需要一些数据，所以也要到服务器分析日志，很麻烦。
对于日志来说，最常见的需求就是收集、存储、查询、展示，开源社区正好有相对应的开源项目：logstash（收集）、elasticsearch（存储+搜索）、kibana（展示），我们将这三个组合起来的技术称之为ELKStack，所以说ELKStack指的是Elasticsearch、Logstash、Kibana技术栈的结合。
ELK Stack （5.0版本之后）Elastic Stack == （ELK Stack + Beats）
ELK Stack包含：ElasticSearch、Logstash、Kibana
ElasticSearch是一个搜索引擎，用来搜索、分析、存储日志。它是分布式的，也就是说可以横向扩容，可以自动发现，索引自动分片，总之很强大。
Logstash用来收集日志，把日志解析为json格式交给ElasticSearch。
Kibana是一个数据可视化组件，把处理后的结果通过web界面展示
Beats在这里是一个轻量级日志收集器，其实Beats家族有5个成员(不断增加) 早期的ELK架构中使用Logstash收集、解析日志，但是Logstash对内存、cpu、io等资源消耗比较高。相比 Logstash，Beats所占系统的CPU和内存几乎可以忽略不计。 x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，是收费的；
ELK的架构图
二、ELK安装准备工作
构建集群的机器角色的分配：
IPhostname节点定义openjdkelasticsearchkibanalogstashbeats192.168.112.150ying04主节点安装安装安装————192.168.112.151ying05数据节点安装安装——安装——192.168.112.152ying06数据节点安装安装————安装
把三台主机的IP 域名增加到/etc/hosts
[root@ying04 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.112.150 ying04
192.168.112.151 ying05
192.168.112.152 ying06三台机器需要安装openjdk；
[root@ying04 ~]# java -version //查无openjdk
-bash: java: 未找到命令
[root@ying06 ~]# yum install -y java-1.8.0-openjdk //安装openjdk
[root@ying04 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)三、配置elasticsearch
三台机器都安装elasticsearch
按照官方文档；创建创建yum源的elastic文件；然后就可以yum安装elasticsearch
[root@ying04 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch //导入官方秘钥
[root@ying04 ~]# vim /etc/yum.repos.d/elastic.repo //创建yum源的elastic文件
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md[root@ying04 ~]# yum install -y elasticsearch
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch
验证中 : elasticsearch-6.4.2-1.noarch 1/1
已安装:
elasticsearch.noarch 0:6.4.2-1
完毕！
[root@ying04 ~]# echo $?
0ying04机器上的配置文件按如下编辑：
[root@ying04 ~]# vim /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
#cluster.name: my-application
cluster.name: fengstory //集群的名称
# ------------------------------------ Node ------------------------------------
#node.name: node-1
node.name: ying04 //节点的名称
#node.attr.rack: r1
node.master: true //此机器是master节点
node.date: false //此机器不是数据节点
# ---------------------------------- Network -----------------------------------
#network.host: 192.168.0.1
network.host: 192.168.112.150 //绑定的IP，意思在哪个IP上监听
#http.port: 9200
http.port: 9200 //端口
# --------------------------------- Discovery ----------------------------------
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["feng04", "feng05", "feng06"] //参与集群的角色ying05机器上的配置文件按如下编辑：
[root@ying05 ~]# vim /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
#cluster.name: my-application
cluster.name: fengstory //集群的名称
# ------------------------------------ Node ------------------------------------
#node.name: node-1
node.name: ying05 //节点的名称
#node.attr.rack: r1
node.master: false //ying05不是master节点
node.date: true //ying05是数据节点，多以为true
# ---------------------------------- Network -----------------------------------
#network.host: 192.168.0.1
network.host: 192.168.112.151 //绑定的IP，意思在哪个IP上监听
#http.port: 9200
http.port: 9200 //端口
# --------------------------------- Discovery ----------------------------------
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["feng04", "feng05", "feng06"] //参与集群的角色ying06机器上的配置文件按如下编辑：
[root@ying06 ~]# vim /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
#cluster.name: my-application
cluster.name: fengstory //集群的名称
# ------------------------------------ Node ------------------------------------
#node.name: node-1
node.name: ying06 //节点的名称
#node.attr.rack: r1
node.master: false //ying05不是master节点
node.date: true //ying05是数据节点，多以为true
# ---------------------------------- Network -----------------------------------
#network.host: 192.168.0.1
network.host: 192.168.112.152 //绑定的IP，意思在哪个IP上监听
#http.port: 9200
http.port: 9200 //端口
# --------------------------------- Discovery ----------------------------------
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["feng04", "feng05", "feng06"] //参与集群的角色三台机器都开启elasticsearch服务
[root@ying04 ~]# systemctl start elasticsearch此时安装配置正确的情况下，每台机器其端口会出现9200和9300
[root@ying04 ~]# ps aux |grep elastic
elastic+ 1163 79.7 28.6 1550360 538184 ? Ssl 17:05 0:03 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.uXgx3jDC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root 1207 0.0 0.0 112720 984 pts/0 R+ 17:05 0:00 grep --color=auto elastic
[root@ying04 ~]# netstat -lnpt |grep java
tcp6 0 0 192.168.112.150:9200 :::* LISTEN 1163/java
tcp6 0 0 192.168.112.150:9300 :::* LISTEN 1163/java四、curl查看elasticsearch
集群健康检查：
[root@ying04 ~]# curl '192.168.112.150:9200/_cluster/health?pretty'
{
"cluster_name" : "fengstory",
"status" : "green", //green 表示正确
"timed_out" : false, //未超时
"number_of_nodes" : 3, //节点3个
"number_of_data_nodes" : 2, //数据节点2个
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}查看集群详细信息：curl '192.168.112.150:9200/_cluster/state?pretty'
[root@ying04 ~]# curl '192.168.112.150:9200/_cluster/state?pretty' |head //集群详细信息，只显示前10行
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
"cluster_name" : "fengstory", //集群名字
"compressed_size_in_bytes" : 9577, //文件太大，约几千行
"cluster_uuid" : "5pI8vvn0RXWBmGmj7Lj54A",
"version" : 5,
"state_uuid" : "Mc803-QnRQ-pkw4UWC7Gqw",
"master_node" : "0nBvsj3DTTmYSdGyiI1obg",
"blocks" : { },
"nodes" : {
"-gLGPb6tTEecUPPhlUlUuA" : {
62 126k 62 81830 0 0 498k 0 --:--:-- --:--:-- --:--:-- 502k
curl: (23) Failed writing body (90 != 16384)五、安装kibana
注意：kibana只需要在ying04 （192.168112.150）上安装
因为已经创建yum源，因此直接用yum安装kibana
[root@ying04 ~]# yum install -y kibana编辑kibana的配置文件：/etc/kibana/kibana.yml
[root@ying04 ~]# vim /etc/kibana/kibana.yml //添加以下配置；带#的是系统举例；为了清晰规整，则需要配置的内容，写在相应的说明下；
#server.port: 5601
server.port: 5601
#server.host: "localhost"
server.host: 192.168.112.150 //只监控主机 150
#elasticsearch.url: "http://localhost:9200"
elasticsearch.url: "http://192.168.112.150:9200"
#logging.dest: stdout
logging.dest: /var/log/kibana.log //定义日志的路径因为kibana的配置文件中，指定日志的文件，因此需要创建此文件，并给予777权限；
[root@ying04 ~]# touch /var/log/kibana.log; chmod 777 /var/log/kibana.log
[root@ying04 ~]# ls -l /var/log/kibana.log
-rwxrwxrwx 1 root root 10075 10月 13 18:25 /var/log/kibana.log启动kibana服务，并查看是否有进程以及5601端口；
[root@ying04 ~]# systemctl start kibana
[root@ying04 ~]# ps aux |grep kibana
kibana 1968 25.2 6.8 1076360 128712 ? Rsl 18:24 0:06 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root 1980 5.0 0.0 112720 984 pts/0 R+ 18:24 0:00 grep --color=auto kibana
[root@ying04 ~]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 536/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 966/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 820/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1031/master
tcp 0 0 192.168.112.150:5601 0.0.0.0:* LISTEN 1968/node //出现5601端口
tcp6 0 0 :::111 :::* LISTEN 536/rpcbind
tcp6 0 0 192.168.112.150:9200 :::* LISTEN 1870/java
tcp6 0 0 192.168.112.150:9300 :::* LISTEN 1870/java
tcp6 0 0 :::22 :::* LISTEN 820/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1031/master此时在浏览器里，访问http://192.168.112.150:5601
能够显示以上页面，说明配置成功；
六、安装logstash
只需要在ying05（192.168.112.151）机器上安装；
[root@ying05 ~]# yum install -y logstash在/etc/logstash/conf.d/目录下，专门存放收集相关定义日志的配置文件
[root@ying05 ~]# ls /etc/logstash/
conf.d jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options
[root@ying05 ~]# ls /etc/logstash/conf.d/ //自定义的配置文件，放在此目录下，才会被加载先创建syslog.conf，专门来收集系统日志；
[root@ying05 ~]# vim /etc/logstash/conf.d/syslog.conf
input { //输入配置
syslog {
type => "system-syslog" //定义日志类型
port => 10514 //定义端口
}
}
output { //输出配置
stdout { //标准输出
codec => rubydebug
}
}检测一下配置是否正确，显示OK，则配置正确；
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
释义：
- --path.settings /etc/logstash/ ：指定logstash配置文件的目录；系统会按照这个目录寻找；
- -f /etc/logstash/conf.d/syslog.conf ：自定义配置与logstash相关的文件；
- --config.test_and_exit ：检测此文件；如果不要exit，则不能自动退出，会直接启动logstash；
[root@ying05 ~]# cd /usr/share/logstash/bin
[root@ying05 bin]# ls
benchmark.sh dependencies-report logstash logstash-keystore logstash.lib.sh logstash-plugin.bat pqrepair setup.bat
cpdump ingest-convert.sh logstash.bat logstash-keystore.bat logstash-plugin pqcheck ruby system-install
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-13T19:06:58,327][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2018-10-13T19:06:58,337][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2018-10-13T19:06:58,942][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-10-13T19:07:01,595][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash修改此rsyslog的配置文件
[root@ying05 bin]# vim /etc/rsyslog.conf
#### RULES ####
*.* @@192.168.112.150:10514 //*.*，表明所有的日志；重启rsyslog
[root@ying05 bin]# systemctl restart rsyslog现在启动logstash，此时会把 所定义的日志全部打印到屏幕上，而且不会自动退出来；相当于开了另一个终端
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-13T22:41:00,835][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"5039884c-a106-4370-8bb3-fcab8227a8d6", :path=>"/var/lib/logstash/uuid"}
[2018-10-13T22:41:01,662][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-13T22:41:05,042][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-13T22:41:05,838][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x9b429e run>"}
[2018-10-13T22:41:06,101][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-10-13T22:41:06,179][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T22:41:06,209][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T22:41:06,757][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-10-13T22:41:17,106][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.112.151:60140"}
{
"logsource" => "ying05",
"message" => "DHCPDISCOVER on ens37 to 255.255.255.255 port 67 interval 19 (xid=0x3a663c52)\n",
"@timestamp" => 2018-10-13T14:41:16.000Z,
"severity_label" => "Informational",
"priority" => 30,
"severity" => 6,
"host" => "192.168.112.151",
"pid" => "2163",
"facility" => 3,
"program" => "dhclient",
"type" => "system-syslog",
"timestamp" => "Oct 13 22:41:16",
"facility_label" => "system",
"@version" => "1"
}
{
"logsource" => "ying05",
"message" => "<info> [1539441739.5305] device (ens37): state change: failed -> disconnected (reason 'none') [120 30 0]\n",
"@timestamp" => 2018-10-13T14:42:19.000Z,
"severity_label" => "Informational",
"priority" => 30,
"severity" => 6,
"host" => "192.168.112.151",
"pid" => "559",
"facility" => 3,
"program" => "NetworkManager",
"type" => "system-syslog",
"timestamp" => "Oct 13 22:42:19",
"facility_label" => "system",
"@version" => "1"
}由于此处是终端显示，不能够退出；会不停的刷新日志情况；此时需要查看其主机的信息，需要复制一个终端（简称B终端）；
在B重点,查看端口，此时有10514；
[root@ying05 ~]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master
tcp6 0 0 :::111 :::* LISTEN 550/rpcbind
tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java
tcp6 0 0 :::10514 :::* LISTEN 2137/java
tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java
tcp6 0 0 :::22 :::* LISTEN 821/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1041/master
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 2137/java回到之前的终端（成为A端口），Ctrl + C强制退出，此时再查看监听端口，10514也不会存在；
^C[2018-10-13T23:52:23,187][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2018-10-13T23:52:23,498][INFO ][logstash.inputs.syslog ] connection error: stream closed
[2018-10-13T23:52:23,651][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x671ad8a0 run>"}
[root@ying05 bin]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master
tcp6 0 0 :::111 :::* LISTEN 550/rpcbind
tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java
tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java
tcp6 0 0 :::22 :::* LISTEN 821/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1041/master此时在A端口上，再次启动logstash；
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f //此启动方式为，前台启动
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-13T23:54:27,377][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-13T23:54:30,556][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-13T23:54:31,118][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7bcbeded run>"}
[2018-10-13T23:54:31,182][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-10-13T23:54:31,217][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T23:54:31,243][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T23:54:31,525][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-10-13T23:58:47,450][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.112.151:60152"}
[2018-10-13T23:58:47,785][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.112.151:60154"}
{
"facility" => 3,
"severity_label" => "Informational",
"program" => "systemd",
"timestamp" => "Oct 13 23:58:47",
"@timestamp" => 2018-10-13T15:58:47.000Z,
"type" => "system-syslog",
"logsource" => "ying05",
"message" => "Stopping System Logging Service...\n",
"severity" => 6,
"facility_label" => "system",
"priority" => 30,
"host" => "192.168.112.151",
"@version" => "1"
}在B终段上，查看端口，一直没有10514，之后出现，说明之前一直在加载，启动后，就会监听10514端口
[root@ying05 ~]# netstat -lnpt |grep 10514
[root@ying05 ~]# netstat -lnpt |grep 10514
[root@ying05 ~]# netstat -lnpt |grep 10514
[root@ying05 ~]# netstat -lnpt |grep 10514
tcp6 0 0 :::10514 :::* LISTEN 2535/java
[root@ying05 ~]#七、配置logstash
刚才只是把日志显示在屏幕上，方便测试，现在需要把日志输入到elasticsearch;
现编辑配置文件syslog.conf
[root@ying05 bin]# vim /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.112.150:9200"] //指向master机器ying04，由于是分布式的可以指向151、152
index => "system-syslog-%{+YYYY.MM}" //定义日志索引
}
}检测配置文件否正常；输出OK则为配置成功；
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-14T00:16:21,163][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-10-14T00:16:23,242][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash现在启动服务
[root@ying05 bin]# systemctl start logstash但是查看日志，却久久不能够被写入，因为权限问题
[root@ying05 bin]# ls -l /var/log/logstash/logstash-plain.log
-rw-r--r-- 1 root root 624 10月 14 00:16 /var/log/logstash/logstash-plain.log
[root@ying05 bin]# chown logstash /var/log/logstash/logstash-plain.log
[root@ying05 bin]# ls -l /var/log/logstash/logstash-plain.log
-rw-r--r-- 1 logstash root 624 10月 14 00:16 /var/log/logstash/logstash-plain.log
[root@ying05 bin]# ls -l /var/lib/logstash/ //这个也需要更改
总用量 4
drwxr-xr-x 2 root root 6 10月 13 19:06 dead_letter_queue
drwxr-xr-x 2 root root 6 10月 13 19:06 queue
-rw-r--r-- 1 root root 36 10月 13 22:41 uuid
[root@ying05 bin]# chown -R logstash /var/lib/logstash/
[root@ying05 bin]# systemctl restart logstash同时9600端口监听，改为192.168.112.151
[root@ying05 bin]# vim /etc/logstash/logstash.yml
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
http.host: "192.168.112.151" //添加主机IP
#此时已经出现10514端口和9600，说明启动成功
[root@ying05 bin]# systemctl restart logstash //重启logstash服务；
[root@ying05 bin]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master
tcp6 0 0 :::111 :::* LISTEN 550/rpcbind
tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java
tcp6 0 0 :::10514 :::* LISTEN 4828/java
tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java
tcp6 0 0 :::22 :::* LISTEN 821/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1041/master
tcp6 0 0 192.168.112.151:9600 :::* LISTEN 4828/java到master机器（ying04）上，查看logstash收集的日志传到能够传到elasticsearch中；现查看其日志索引，可以看到system-syslog-2018.10
[root@ying04 ~]# curl '192.168.112.150:9200/_cat/indices?v' //看到索引，说明logstash与elasticsearch通信正常
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open system-syslog-2018.10 uP2TM4UFTdSx7fbvLD1IsQ 5 1 82 0 773.8kb 361.9kb现在我们从feng06（192.168.112.152）机器，由ssh登录到feng05（192.168.112.151）机器，会产生日志；
[root@ying06 ~]# ssh 192.168.112.151
The authenticity of host '192.168.112.151 (192.168.112.151)' can't be established.
ECDSA key fingerprint is SHA256:ZQlXi+kieRwi2t64Yc5vUhPPWkMub8f0CBjnYRlX2Iw.
ECDSA key fingerprint is MD5:ff:9f:37:87:81:89:fc:ed:af:c6:62:c6:32:53:7a:ad.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.112.151' (ECDSA) to the list of known hosts.
root@192.168.112.151's password:
Last login: Sun Oct 14 13:55:30 2018 from 192.168.112.1在feng05机器上查看，这个时间段发生的日志；
[root@ying05 ~]# less /var/log/messages
......截取14.25.04这个时间点
Oct 14 14:25:04 ying05 systemd: Started Session 14 of user root.
Oct 14 14:25:04 ying05 systemd-logind: New session 14 of user root.
Oct 14 14:25:04 ying05 systemd: Starting Session 14 of user root.在浏览器上，刷新kibana；会出现日志，就是虚拟机显示的日志；注意时间点的对照
八、收集nginx日志
在/etc/logstash/conf.d/目录下创建一个收集nginx的配置文件；
[root@ying05 ~]# cd /etc/logstash/conf.d/
[root@ying05 conf.d]# ls
syslog.conf
[root@ying05 conf.d]# vim nginx.conf //创建一个收集nginx的配置文件
input {
file {
path => "/tmp/elk_access.log" //产生日志的路径
start_position => "beginning" //从哪里开始
type => "nginx" //类型标记
}
}
filter { //有grok过滤器进行解析字段
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.112.151:9200"] //主机ying05
index => "nginx-test-%{+YYYY.MM.dd}"
}
}对此配置进行检查；生成OK，就是正确的；
[root@ying05 conf]# cd /usr/share/logstash/bin
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-15T08:31:42,427][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-10-15T08:31:47,080][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash需要你的机器上安装nginx；下面得知，nginx已经运行，而且也在占用80端口，OK；
[root@ying05 ~]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master
tcp6 0 0 :::111 :::* LISTEN 550/rpcbind
tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java
tcp6 0 0 :::10514 :::* LISTEN 4828/java
tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java
tcp6 0 0 :::22 :::* LISTEN 821/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1041/master
tcp6 0 0 192.168.112.151:9600 :::* LISTEN 4828/java在nginx配置文件中，添加代理kibana（安装在feng04的主机）的虚拟主机文件；
[root@ying05 ~]# cd /usr/local/nginx/conf/
[root@ying05 conf]# vim nginx.conf
server {
listen 80;
server_name elk.ying.com;
location / {
proxy_pass http://192.168.112.150:5601; //真实服务器
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2; //定义日志格式为main2
}由于定义的格式为main2，因此需要在nginx配置文件中也需要 添加main2格式
[root@ying05 conf]# vim nginx.conf //跟上面为同一个配置
......默认
log_format combined_realip '$remote_addr $http_x_forwarded_for [$time_local]'
' $host "$request_uri" $status'
' "$http_referer" "$http_user_agent"'; //此为默认配置
...... 默认
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$upstream_addr" $request_time'; //此为main2格式定义重启nginx服务；
[root@ying05 conf]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@ying05 conf]# /usr/local/nginx/sbin/nginx -s reload再到 C:\Windows\System32\drivers\etc目录下，hosts下，配置定义的域名；
192.168.112.151 elk.ying.com再到ying05上，重启logstash 服务
[root@ying05 conf]# systemctl restart logstash稍等片刻后，到ying04机器上，查看生成的索引 nginx-test-2018.10.15；
[root@ying04 ~]# curl '192.168.112.150:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana aO3JiaT_TKWt3OJhDjPOvg 1 0 3 0 17.8kb 17.8kb
yellow open nginx-test-2018.10.15 taXOvQTyTFely-_oiU_Y2w 5 1 60572 0 6mb 6mb
yellow open system-syslog-2018.10 uP2TM4UFTdSx7fbvLD1IsQ 5 1 69286 0 10.7mb 10.7mb到浏览器上，输入elk.ying.com；并点击 index patterns
把nginx-test-2018.10.15 这个索引填入到图示位置；
填写好之后，点击 creat，就可以看到 success! 添加索引成功；点击下一步；
此时点击左侧Management，会出现两个索引，其中一个就是刚才添加的 nginx-test-2018.10.15;
在ying05机器上查看/tmp/elk_access.log日志，9.15分的日志，与kibana显示的对比；结果肯定是一样的；
[root@ying05 bin]# less /tmp/elk_access.log
elk.ying.com 192.168.112.1 - - [15/Oct/2018:09:05:18 +0800] "POST /elasticsearch/_msearch HTTP/1.1" 200 5919 "http://elk.ying.com/app/kibana" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" "192.168.112.150:5601" 1.287请看下面的message；注意时间是否一致；测试成功；
九、使用beats采集日志
目前还有一种日志收集器beats；
- beats是轻量级采集日志，耗用资源少，可扩展；
- logstash耗费资源；
先yum安装filebeat
[root@ying06 ~]yum list |grep filebeat
filebeat.x86_64 6.4.2-1 @elasticsearch-6.x
filebeat.i686 6.4.2-1 elasticsearch-6.x
[root@ying06 ~]yum install -y filebeat按下面编辑filebeat.yml的配置文件，目的是为了测试，此配置；
[root@ying06 ~]# vim /etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
# Change to true to enable this input configuration.
# enabled: false //默认范例
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log //默认范例
- /var/log/messages //指定日志路径
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"] //范例
output.console:
enable: true此时会显示很对的日志，不断刷屏，此时显示的日志，就是"source":"/var/log/messages"
[root@ying06 ~]/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
{"@timestamp":"2018-10-15T07:32:06.322Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"beat":{"version":"6.4.2","name":"ying06","hostname":"ying06"},"host":{"name":"ying06"},"source":"/var/log/messages","offset":1253647,"message":"Oct 15 15:32:04 ying06 NetworkManager[558]: \u003cwarn\u003e [1539588724.3946] device (ens37): Activation: failed for connection '有线连接 1'","prospector":{"type":"log"},"input":{"type":"log"}}
{"@timestamp":"2018-10-15T07:32:06.322Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"host":{"name":"ying06"},"source":"/var/log/messages","offset":1253784,"message":"Oct 15 15:32:04 ying06 NetworkManager[558]: \u003cinfo\u003e [1539588724.3958] device (ens37): state change: failed -\u003e disconnected (reason 'none') [120 30 0]","prospector":{"type":"log"},"input":{"type":"log"},"beat":{"name":"ying06","hostname":"ying06","version":"6.4.2"}}
^C[root@ying06 ~]#此两个message与上面采集的日志，是对应的；
[root@ying06 ~]# less /var/log/messages
Oct 15 15:32:04 ying06 NetworkManager[558]: <warn> [1539588724.3946] device (ens37): Activation: failed for connection '有线连接 1'
Oct 15 15:32:04 ying06 NetworkManager[558]: <info> [1539588724.3958] device (ens37): state change: failed -> disconnected (reason 'none') [120 30 0]现在只是把日志显示在屏幕上，现在需要把日志，传到elasticsearch，再通过kibana可视化显示；
[root@ying06 ~]# vim /etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- /var/log/elasticsearch/fengstory.log //现在收集的此日志
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch: //输出到elasticsearch中
# Array of hosts to connect to.
hosts: ["192.168.112.150:9200"] //指向feng04主机
#output.console: //之前测试的 不用加载
# enable: true开启filebeat服务，并查看其进程；
[root@ying06 ~]# systemctl start filebeat
[root@ying06 ~]# ps aux|grep filebeat
root 1599 0.0 0.8 309872 16528 ? Ssl 16:20 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root 1704 0.0 0.0 112720 980 pts/0 R+ 16:47 0:00 grep --color=auto filebeat
[root@ying06 ~]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 556/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 964/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 827/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1034/master
tcp6 0 0 :::111 :::* LISTEN 556/rpcbind
tcp6 0 0 192.168.112.152:9200 :::* LISTEN 1711/java
tcp6 0 0 192.168.112.152:9300 :::* LISTEN 1711/java
tcp6 0 0 :::22 :::* LISTEN 827/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1034/master在feng04（192.168.112.150）机器上,查看日志是否到达elasticsearch中；出现 filebeat字样，说明已经到达；
[root@ying04 ~]# curl '192.168.112.150:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open nginx-test-2018.10.15 taXOvQTyTFely-_oiU_Y2w 5 1 202961 0 36.1mb 18.5mb
green open .kibana aO3JiaT_TKWt3OJhDjPOvg 1 1 4 0 80.3kb 40.1kb
green open filebeat-6.4.2-2018.10.15 m7Biv3QMTXmRR5u-cxIAoQ 3 1 73 0 153.3kb 95.4kb
green open system-syslog-2018.10 uP2TM4UFTdSx7fbvLD1IsQ 5 1 211675 0 41.9mb 21.4mbfeng04上端口也有5601,此时可以在浏览器上查看；
[root@ying04 ~]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 543/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 907/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 820/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1042/master
tcp 0 0 192.168.112.150:5601 0.0.0.0:* LISTEN 1420/node
tcp6 0 0 :::111 :::* LISTEN 543/rpcbind
tcp6 0 0 192.168.112.150:9200 :::* LISTEN 1255/java
tcp6 0 0 192.168.112.150:9300 :::* LISTEN 1255/java
tcp6 0 0 :::22 :::* LISTEN 820/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1042/master此时在浏览器访问：elk.ying.com
创建成功后，就可以看到日志 /var/log/elasticsearch/fengstory.log
此时查看日志来源，通过对比，可以得出两者一样
[root@ying06 ~]# less /var/log/elasticsearch/fengstory.log
[2018-10-15T16:49:08,548][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [ying06] publish_address {192.168.112.152:9200}, bound_addresses {192.168.112.152:9200}
[2018-10-15T16:49:08,548][INFO ][o.e.n.Node ] [ying06] started
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
