1、截至目前Elasticsearch 版本已经更新到了7.10.1版本了，这里先使用Filebeat 6.7.1版本，给一个下载地址，如下所示：
官方下载地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch" target="_blank">https://www.elastic.co/cn/downloads/past-releases#elasticsearch
2、Beats入门学习。Lightweight Data Shipper，轻量级的数据传送者。
1）、Filebeat，主要用于日志文件处理。处理流程：输入Input、处理Filter、输出Output。
2）、Metricbeat，主要用来度量数据。要用来搜集cpu数据，内存数据，磁盘数据，nginx，mysql。
3）、Packetbeat，主要用来处理网络数据，主要针对网络数据，可以抓包进行分析。
4）、Winlogbeat，主要用来处理Windows数据。
5）、Auditbeat。
6）、Heartbeat建康检查。
7）、Functionbeat。
3、Filebeat，主要用于对日志文件进行处理，Filebeat的配置简介：
1）、Filebeat Input配置简介，使用的是yaml语法。input_type目前有两个类型，分别是log日志文件、stdin标准输入。
1 案例如下所示：
2 filebeat.prospectors:
3 -input_type:log # 日志文件，有两个类型，分别是log日志文件、stdin标准输入。
4 paths: # 日志文件路径
5 -/var/log/apache/httpd-*.log
6 -input_type:log
7 paths:
8 -/var/log/messages
9 -/var/log/*.log
2）、Filebeat Output配置简介，支持的Output对象包含，Console标准输出、Elasticsearch、Logstash、Kafka、Redis、File文件。
1 案例如下所示：
2 output.elasticsearch:
3 hosts:["http://localhost:9200"] # 案例一，elasticsearch的连接地址
4 username:"admin" # 用户权限认证，需要配置账号密码
5 password:"123456"
6
7 output.console: # 案例二，输出到控制台，方便调试。
8 pretty:true # 输出做json的格式化。
3）、Filebeat Filter配置简介，Filbeat可以对输入内容进行一定处理的，包含Input时处理、Output前处理，如下所示：
a）、Input 时处理。
Include_lines : 达到某些条件的时候，读入这一行。
exclude_lines ：达到某些条件的时候，不读入这一行。
exclude_files：当文件名符合某些条件的时候，不读取这个文件。
b）、output 前处理 --Processor。
drop_event ：读取到某一条，满足了某个条件，不输出。
drop_fields ：读取到某一条，满足了某个条件，不输出这个字段。
Decode_json_fields ：把这条数据里面符合json格式的字段，去做json的解析。
Include_fields ：加入一些字段，或者是只想取数据里面的某一些字段。
1 案例如下所示:
2 processors: # 案例一
3 -drop_event:
4 when:
5 regexp: # 正则表达式，当匹配到message字段以DBG开头的进行丢弃。
6 message:"^DBG:"
7
8 processors: # 案例二
9 -decode_json_fields: # 将结果处理成正常的json格式的。
10 fields:["inner"]
4）、Filebeat高级使用简介：
Filebeat + Elasticsearch Ingest Node的组合使用，原因是因为Filebeat 缺乏数据转换的能力。
Elasticsearch Ingest Node介绍如下所示，新增的node类型，在数据写入es前对数据进行处理转换，使用的api是pipeline api。
4、将下载好的Filebeat上传到服务器，然后进行解压缩，然后将解压缩后的包授权给elsearch用户，如下所示：
1 [root@k8s-master package]# tar -zxvf filebeat-6.7.1-linux-x86_64.tar.gz -C /usr/local/elastic/
由于使用的是root进行解压缩操作的，将文件拥有者赋予给elsearch自己新建的用户和用户组。
1 [root@k8s-master elastic]# ll
2 total 0
3 drwxr-xr-x 9 elsearch elsearch 155 Jan 9 23:08 elasticsearch-6.7.1
4 drwxr-xr-x 5 root root 212 Jan 10 19:27 filebeat-6.7.1-linux-x86_64
5 drwxr-xr-x 13 elsearch elsearch 263 Jan 9 23:41 kibana-6.7.1-linux-x86_64
6 [root@k8s-master elastic]# chown -R elsearch:elsearch filebeat-6.7.1-linux-x86_64/
7 [root@k8s-master elastic]# ll
8 total 0
9 drwxr-xr-x 9 elsearch elsearch 155 Jan 9 23:08 elasticsearch-6.7.1
10 drwxr-xr-x 5 elsearch elsearch 212 Jan 10 19:27 filebeat-6.7.1-linux-x86_64
11 drwxr-xr-x 13 elsearch elsearch 263 Jan 9 23:41 kibana-6.7.1-linux-x86_64
12 [root@k8s-master elastic]# cd filebeat-6.7.1-linux-x86_64/
13 [root@k8s-master filebeat-6.7.1-linux-x86_64]# ll
14 total 36700
15 -rw-r--r-- 1 elsearch elsearch 146747 Apr 2 2019 fields.yml
16 -rwxr-xr-x 1 elsearch elsearch 37162115 Apr 2 2019 filebeat
17 -rw-r--r-- 1 elsearch elsearch 69996 Apr 2 2019 filebeat.reference.yml
18 -rw------- 1 elsearch elsearch 7714 Apr 2 2019 filebeat.yml
19 drwxr-xr-x 4 elsearch elsearch 24 Apr 2 2019 kibana
20 -rw-r--r-- 1 elsearch elsearch 13675 Apr 2 2019 LICENSE.txt
21 drwxr-xr-x 21 elsearch elsearch 287 Apr 2 2019 module
22 drwxr-xr-x 2 elsearch elsearch 4096 Apr 2 2019 modules.d
23 -rw-r--r-- 1 elsearch elsearch 163444 Apr 2 2019 NOTICE.txt
24 -rw-r--r-- 1 elsearch elsearch 802 Apr 2 2019 README.md
25 [root@k8s-master filebeat-6.7.1-linux-x86_64]#
其中，进入到Filebeat的目录，里面的文件介绍，如下所示：
a）、data存储的是filebeat解析过程中会去存日志读到的位置。
b）、filebeat是可执行文件。
c）、module是filebeat支持的模块的功能，filebeat.yml是参考配置。
5、通过一个简单案例，使用Filebeat收集nginx的log日志，通过stdin收集日志。通过console输出结果。
注意：nginx的默认日志路径为/var/log/nginx/。
1 [elsearch@k8s-master elastic]$ cd filebeat-6.7.1-linux-x86_64/
2 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ ls
3 fields.yml filebeat filebeat.reference.yml filebeat.yml kibana LICENSE.txt module modules.d NOTICE.txt README.md
4 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ ll
5 total 36700
6 -rw-r--r-- 1 elsearch elsearch 146747 Apr 2 2019 fields.yml
7 -rwxr-xr-x 1 elsearch elsearch 37162115 Apr 2 2019 filebeat
8 -rw-r--r-- 1 elsearch elsearch 69996 Apr 2 2019 filebeat.reference.yml
9 -rw------- 1 elsearch elsearch 7714 Apr 2 2019 filebeat.yml
10 drwxr-xr-x 4 elsearch elsearch 24 Apr 2 2019 kibana
11 -rw-r--r-- 1 elsearch elsearch 13675 Apr 2 2019 LICENSE.txt
12 drwxr-xr-x 21 elsearch elsearch 287 Apr 2 2019 module
13 drwxr-xr-x 2 elsearch elsearch 4096 Apr 2 2019 modules.d
14 -rw-r--r-- 1 elsearch elsearch 163444 Apr 2 2019 NOTICE.txt
15 -rw-r--r-- 1 elsearch elsearch 802 Apr 2 2019 README.md
16 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ cp filebeat.yml nginx.yml
17 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ ll
18 total 36708
19 -rw-r--r-- 1 elsearch elsearch 146747 Apr 2 2019 fields.yml
20 -rwxr-xr-x 1 elsearch elsearch 37162115 Apr 2 2019 filebeat
21 -rw-r--r-- 1 elsearch elsearch 69996 Apr 2 2019 filebeat.reference.yml
22 -rw------- 1 elsearch elsearch 7714 Apr 2 2019 filebeat.yml
23 drwxr-xr-x 4 elsearch elsearch 24 Apr 2 2019 kibana
24 -rw-r--r-- 1 elsearch elsearch 13675 Apr 2 2019 LICENSE.txt
25 drwxr-xr-x 21 elsearch elsearch 287 Apr 2 2019 module
26 drwxr-xr-x 2 elsearch elsearch 4096 Apr 2 2019 modules.d
27 -rw------- 1 elsearch elsearch 7714 Jan 10 19:50 nginx.yml
28 -rw-r--r-- 1 elsearch elsearch 163444 Apr 2 2019 NOTICE.txt
29 -rw-r--r-- 1 elsearch elsearch 802 Apr 2 2019 README.md
30 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$
这里我直接复制了一份filebeat.yml文件，然后重命名为nginx.yml。
1 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$
2 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ head -n 10 /var/log/nginx/access.log
3 192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"
4 192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET /favicon.ico HTTP/1.1" 404 570 "http://192.168.110.133/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"
5 192.168.110.33 - - [10/Jan/2021:19:46:04 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
6 192.168.110.33 - - [10/Jan/2021:19:46:04 +0800] "GET /favicon.ico HTTP/1.1" 404 570 "http://192.168.110.133/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
7 192.168.110.33 - - [10/Jan/2021:19:46:23 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
8 192.168.110.33 - - [10/Jan/2021:19:46:23 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
9 192.168.110.33 - - [10/Jan/2021:19:46:23 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
10 192.168.110.33 - - [10/Jan/2021:19:46:50 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
11 192.168.110.33 - - [10/Jan/2021:19:46:50 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
12 192.168.110.33 - - [10/Jan/2021:19:46:51 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
13 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$
14 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ ls
15 fields.yml filebeat filebeat.reference.yml filebeat.yml kibana LICENSE.txt module modules.d nginx.yml NOTICE.txt README.md
16 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ vim nginx.yml
17 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ vim nginx.yml
18 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ head -n 2 /var/log/nginx/access.log | ./filebeat -e -c nginx.yml
运行之后，可以看到，nginx的日志，已经在控制台进行输出了，如下所示：
其中，nginx.yml的配置如下所示：
1 [elsearch@k8s-master filebeat-6.7.1-linux-x86_64]$ cat nginx.yml
2 ###################### Filebeat Configuration Example #########################
3
4 # This file is an example configuration file highlighting only the most common
5 # options. The filebeat.reference.yml file from the same directory contains all the
6 # supported options with more comments. You can use it as a reference.
7 #
8 # You can find the full configuration reference here:
9 # https://www.elastic.co/guide/en/beats/filebeat/index.html
10
11 # For more available modules and options, please see the filebeat.reference.yml sample
12 # configuration file.
13
14 #=========================== Filebeat inputs =============================
15
16 filebeat.inputs:
17
18 # Each - is an input. Most options can be set at the input level, so
19 # you can use different inputs for various configurations.
20 # Below are the input specific configurations.
21
22 - type: stdin
23
24 # Change to true to enable this input configuration.
25 # enabled: false
26
27 # Paths that should be crawled and fetched. Glob based paths.
28 # paths:
29 # - /var/log/*.log
30 #- c:\programdata\elasticsearch\logs\*
31
32 # Exclude lines. A list of regular expressions to match. It drops the lines that are
33 # matching any regular expression from the list.
34 #exclude_lines: ['^DBG']
35
36 # Include lines. A list of regular expressions to match. It exports the lines that are
37 # matching any regular expression from the list.
38 #include_lines: ['^ERR', '^WARN']
39
40 # Exclude files. A list of regular expressions to match. Filebeat drops the files that
41 # are matching any regular expression from the list. By default, no files are dropped.
42 #exclude_files: ['.gz$']
43
44 # Optional additional fields. These fields can be freely picked
45 # to add additional information to the crawled log files for filtering
46 #fields:
47 # level: debug
48 # review: 1
49
50 ### Multiline options
51
52 # Multiline can be used for log messages spanning multiple lines. This is common
53 # for Java Stack Traces or C-Line Continuation
54
55 # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
56 #multiline.pattern: ^\[
57
58 # Defines if the pattern set under pattern should be negated or not. Default is false.
59 #multiline.negate: false
60
61 # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
62 # that was (not) matched before or after or as long as a pattern is not matched based on negate.
63 # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
64 #multiline.match: after
65
66
67 #============================= Filebeat modules ===============================
68
69 filebeat.config.modules:
70 # Glob pattern for configuration loading
71 path: ${path.config}/modules.d/*.yml
72
73 # Set to true to enable config reloading
74 reload.enabled: false
75
76 # Period on which files under path should be checked for changes
77 #reload.period: 10s
78
79 #==================== Elasticsearch template setting ==========================
80
81 setup.template.settings:
82 index.number_of_shards: 3
83 #index.codec: best_compression
84 #_source.enabled: false
85
86 #================================ General =====================================
87
88 # The name of the shipper that publishes the network data. It can be used to group
89 # all the transactions sent by a single shipper in the web interface.
90 #name:
91
92 # The tags of the shipper are included in their own field with each
93 # transaction published.
94 #tags: ["service-X", "web-tier"]
95
96 # Optional fields that you can specify to add additional information to the
97 # output.
98 #fields:
99 # env: staging
100
101 #================================ Outputs =====================================
102
103 # Configure what output to use when sending the data collected by the beat.
104
105 #-------------------------- Elasticsearch output ------------------------------
106 output.console:
107 pretty: true
108 # Array of hosts to connect to.
109 # hosts: ["192.168.110.133:9200"]
110
111 # Enabled ilm (beta) to use index lifecycle management instead daily indices.
112 #ilm.enabled: false
113
114 # Optional protocol and basic auth credentials.
115 #protocol: "https"
116 #username: "elastic"
117 #password: "changeme"
6、Packetbeat简介，可以实时抓取网络包，自动解析应用层协议（ICMP、DNS、HTTP、Mysql、Redis等等），Wireshark作为抓包工具，而Packetbeat作为轻量级的抓包工具。
Packetbeat解析http协议，比如解析elasticsearch http请求，如下所示：
1 packetbeat.interfaces.device:any # 指定网络的网卡，any指定所有网卡
2 packetbeat.protocols.http: ports: [9200] # 抓http协议的，端口是9200
3 send_request: true # 记录http请求的request内容，默认是false
4 include_body_for: ["application/json","x-www-form-urlencoded"] # 设置两种格式的数据都可以进行记录
5 output.console:
6 pretty: true # 标准控制台输出
将packetbeat下载好上传到服务器进行解压缩，然后将权限授予elsearch自己创建的用户即可:
1 [root@k8s-master package]# tar -zxvf packetbeat-6.7.1-linux-x86_64.tar.gz -C /usr/local/elastic/
然后将权限授予elsearch用户，如下所示：
1 [root@k8s-master elastic]# ll
2 total 0
3 drwxr-xr-x 9 elsearch elsearch 155 Jan 9 23:08 elasticsearch-6.7.1
4 drwxr-xr-x 6 elsearch elsearch 241 Jan 10 20:05 filebeat-6.7.1-linux-x86_64
5 drwxr-xr-x 13 elsearch elsearch 263 Jan 9 23:41 kibana-6.7.1-linux-x86_64
6 drwxr-xr-x 13 elsearch elsearch 267 Jan 10 21:15 logstash-6.7.1
7 drwxr-xr-x 3 root root 187 Jan 10 22:50 packetbeat-6.7.1-linux-x86_64
8 [root@k8s-master elastic]# chown -R elsearch:elsearch packetbeat-6.7.1-linux-x86_64/
9 [root@k8s-master elastic]# ll
10 total 0
11 drwxr-xr-x 9 elsearch elsearch 155 Jan 9 23:08 elasticsearch-6.7.1
12 drwxr-xr-x 6 elsearch elsearch 241 Jan 10 20:05 filebeat-6.7.1-linux-x86_64
13 drwxr-xr-x 13 elsearch elsearch 263 Jan 9 23:41 kibana-6.7.1-linux-x86_64
14 drwxr-xr-x 13 elsearch elsearch 267 Jan 10 21:15 logstash-6.7.1
15 drwxr-xr-x 3 elsearch elsearch 187 Jan 10 22:50 packetbeat-6.7.1-linux-x86_64
16 [root@k8s-master elastic]#
Packetbeat运行命令：sudo ./packetbeat -e -c es.yml -strict.perms=false，抓包时候需要使用root权限，-strict.perms=false不再检查配置文件es.yml的权限，方便运行。
复制一个packetbeat.yml为es.yml，然后修改成，如下所示：
1 #################### Packetbeat Configuration Example #########################
2
3 # This file is an example configuration file highlighting only the most common
4 # options. The packetbeat.full.yml file from the same directory contains all the
5 # supported options with more comments. You can use it as a reference.
6 #
7 # You can find the full configuration reference here:
8 # https://www.elastic.co/guide/en/beats/packetbeat/index.html
9
10 #============================== Network device ================================
11
12 # Select the network interface to sniff the data. On Linux, you can use the
13 # "any" keyword to sniff on all connected interfaces.
14 packetbeat.interfaces.device: any
15
16 packetbeat.protocols.http:
17 # Configure the ports where to listen for HTTP traffic. You can disable
18 # the HTTP protocol by commenting out the list of ports.
19 ports: [9200]
20 send_request: true
21 include_body_for: ["application/json", "x-www-form-urlencoded"]
22
23
24 #================================ Outputs =====================================
25
26 # Configure what outputs to use when sending the data collected by the beat.
27 # Multiple outputs may be used.
28
29 #-------------------------- Elasticsearch output ------------------------------
30 #output.elasticsearch:
31 # Array of hosts to connect to.
32 # hosts: ["localhost:9200"]
33
34 # Optional protocol and basic auth credentials.
35 #protocol: "https"
36 #username: "elastic"
37 #password: "changeme"
38
39 output.console:
40 pretty: true
41
42 #================================ Logging =====================================
43
44 # Sets log level. The default log level is info.
45 # Available log levels are: critical, error, warning, info, debug
46 #logging.level: debug
47
48 # At debug level, you can selectively enable logging only for some components.
49 # To enable all selectors use ["*"]. Examples of other selectors are "beat",
50 # "publish", "service".
51 #logging.selectors: ["*"]
运行启动命令，如果报下面的错误，需要将elsearch用户加到配置文件中，如下所示：
1 [elsearch@k8s-master packetbeat-6.7.1-linux-x86_64]$ sudo ./packetbeat -e -c es.yml -strict.perms=false
2 [sudo] password for elsearch:
3 elsearch is not in the sudoers file. This incident will be reported.
4 [elsearch@k8s-master packetbeat-6.7.1-linux-x86_64]$
解决方法，如下所示：
1 [elsearch@k8s-master packetbeat-6.7.1-linux-x86_64]$ su root
2 Password:
3 [root@k8s-master packetbeat-6.7.1-linux-x86_64]# chmod u+w /etc/sudoers
4 [root@k8s-master packetbeat-6.7.1-linux-x86_64]# vi /etc/sudoers
5 [root@k8s-master packetbeat-6.7.1-linux-x86_64]# vim /etc/sudoers
6 [root@k8s-master packetbeat-6.7.1-linux-x86_64]# chmod u-w /etc/sudoers
7 [root@k8s-master packetbeat-6.7.1-linux-x86_64]#
8 [root@k8s-master packetbeat-6.7.1-linux-x86_64]#
解决方案，步骤：
第一步，切换到root用户下: su root
第二步，添加sudo文件的写权限,命令是: chmod u+w /etc/sudoers
第三步，编辑sudoers文件： vim /etc/sudoers，找到这行 root ALL=(ALL) ALL,在他下面添加xxx ALL=(ALL) ALL (这里的xxx是你的用户名)。这里说下你可以sudoers添加下面四行中任意一条。
elsearch ALL=(ALL) ALL 第一行:允许用户elsearch执行sudo命令(需要输入密码)。
%elsearch ALL=(ALL) ALL 第二行:允许用户组elsearch里面的用户执行sudo命令(需要输入密码)。
elsearch ALL=(ALL) NOPASSWD: ALL 第三行:允许用户elsearch执行sudo命令,并且在执行的时候不输入密码。
%elsearch ALL=(ALL) NOPASSWD: ALL 第四行:允许用户组elsearch里面的用户执行sudo命令,并且在执行的时候不输入密码。
第四步。撤销sudoers文件写权限,命令: chmod u-w /etc/sudoers，这样普通用户就可以使用sudo了。
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
