Logstash：
Logstash是一个具有实时pipeline功能的开源数据收集引擎。Logstash可以动态的统一来自不同数据源的数据，并将数据规范化到你选择的目的地。虽然Logstash最初推动了日志收集方面的创新，但它的功能现在更丰富了。任何类型的事件都可以通过丰富的input，filter，output插件进行转换，简化抽取过程。
环境概况
系统类型：Centos7
elasticsearch:10.211.55.9
logstash:10.211.55.10
kibana:10.211.55.11
步骤一、安装完成logstash
查看上一篇博客：《ELK简单部署》
步骤二、查看插件
# /opt/logstash/bin/logstash-plugin list //查看插件
logstash-input-stdin //标准输入插件
logstash-output-stdout //标准输出插件
...
# vim /etc/logstash/logstash.conf
input{
stdin{
}
}
filter{
}
output{
stdout{
}
}
# alias logstash='/opt/logstash/bin/logstash'
# logstash -f /etc/logstash/logstash.conf //启动并测试
Settings: Default pipeline workers: 2
Pipeline main started
aa //logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
2018-09-15T06:19:28.724Z logstash aa备注：若不会写配置文件可以找帮助，插件文档的位置：
https://github.com/logstash-plugins 帮助文档：
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
步骤三、插件的介绍
1、codec类插件
# vim /etc/logstash/logstash.conf
input{
stdin{ codec => json }
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
dqw //输入一个普通字符串
{
"message" => "dqw",
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"@timestamp" => "2018-12-26T06:48:36.813Z",
"host" => "logstash"
}
{"a":1,"b":2,"c":3} //输入一个json字符串
{
"a" => 1,
"b" => 2,
"c" => 3,
"@version" => "1",
"@timestamp" => "2018-12-26T06:49:10.724Z",
"host" => "logstash"
}2、file模块插件
（1）
# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file{
path => ["/tmp/a.log","/var/tmp/b.log"]
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# touch /tmp/a.log
# touch /var/tmp/b.log
# logstash -f /etc/logstash/logstash.conf
另开一个终端：写入数据
# echo a1 > /tmp/a.log
# echo b1 > /var/tmp/b.log
返回前一个终端查看，能看到输出数据（2）
# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file{
path => ["/tmp/a.log","/var/tmp/b.log"]
sincedb_path => "/var/lib/logstash/since.db" //记录读取文件的位置
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# logstash -f /etc/logstash/logstash.conf
另开一个终端：
# cd
# ls -a
# cat .sincedb_e9a1772295a869da80134b5c4e75816e
//默认保存在root家目录
# rm -rf .sincedb_e9a1772295a869da80134b5c4e75816e
# echo aa > /tmp/a.log
# echo bb > /var/tmp/b.log
返回前一个终端查看，没有输出数据（3）
# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file{
path => ["/tmp/a.log","/var/tmp/b.log"]
sincedb_path => "/var/lib/logstash/since.db"
start_position => "beginning" //配置第一次读取文件从什么地方开始
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# logstash -f /etc/logstash/logstash.conf
//没有数据输出，因为文件since.db在上一步已经创建
# rm -rf /var/lib/logstash/since.db
# logstash -f /etc/logstash/logstash.conf //能看到上一步追加的数据从头开始输出（4）
# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file{
path => ["/tmp/a.log","/var/tmp/b.log"]
sincedb_path => "/var/lib/logstash/since.db"
start_position => "beginning"
type => "testlog" //类型名称
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# rm -rf /var/lib/logstash/since.db
# logstash -f /etc/logstash/logstash.conf //能看到数据有类型（5）
如果两个文件记录的类型不同，可以分开两个file来写
# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file{
path => ["/tmp/a.log"]
sincedb_path => "/var/lib/logstash/since.db"
start_position => "beginning"
type => "testlog"
}
file{
path => ["/var/tmp/b.log"]
sincedb_path => "/var/lib/logstash/since.db"
start_position => "beginning"
type => "tlog"
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}3、tcp、udp模块插件
# vim /etc/logstash/logstash.conf
input
stdin{ codec => "json" }
file{
path => ["/tmp/a.log","/var/tmp/b.log"]
sincedb_path => "/var/lib/logstash/since.db"
start_position => "beginning"
type => "testlog"
}
tcp{
host => "0.0.0.0"
mode => "server"
port => 8888
type => "tcplog"
}
udp{
host => "0.0.0.0"
port => 8888
type => "udplog"
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# logstash -f /etc/logstash/logstash.conf
另开一个终端查看，可以看到端口
# netstat -antup | grep 8888
验证方法1：
kibana主机：
# echo "abcd" > /dev/udp/10.211.5.10/8888
返回启动的logstash主机，查看数据
验证方法2：
kibana主机：
# function send() { //定义函数
> exec 8<>/dev/tcp/10.211.5.10/8888 //8可以任意数字都可以
> echo $1 >&8
> exec 8<&- //tcp结束退出
> }
# send hahaha
logstash主机查看结果4、syslog插件练习
/var/log/messages、/etc/rsyslog.conf
kibana主机：
# logger -p local0.info -t testlog "haha"
# cat /var/log/messages
# vim /etc/rsyslog.conf
local0.info /var/log/mylog //自己添加这一行
# systemctl restart rsyslog //重启rsyslog
# ll /var/log/mylog //提示没有那个文件或目录
# logger -p local0.info -t testlog "haha"
# cat /var/log/mylog
# vim /etc/rsyslog.conf
local0.info @@10.211.5.10:514
# logger -p local0.info -t testlog "xixihaha"
logstash主机：
input{
stdin{ codec => "json" }
file{
path => ["/tmp/a.log","/var/tmp/b.log"]
sincedb_path => "/var/lib/logstash/since.db"
start_position => "beginning"
type => "testlog"
}
tcp{
host => "0.0.0.0"
mode => "server"
port => 8888
type => "tcplog"
}
udp{
host => "0.0.0.0"
port => 8888
type => "udplog"
}
syslog {
port => "514"
type => "syslog"
}
}
filter{
}
output{
stdout{ codec => rubydebug }
}
# logstash -f /etc/logstash/logstash.conf
另开logstash终端：
# netstat -antup | grep 514
kibana主机：
# logger -p local0.info -t testlog "xixihaha"
返回启动的logstash主机，查看数据
rsyslog.conf配置向远程发送数据，远程登陆5.11的时侯，把登陆日志的信息（/var/log/secure）转发给logstash即5.10这台机器
kibana主机：
修改57行
authpriv.* @@10.211.5.10:514
# systemctl restart rsyslog
logstash主机：
# logstash -f /etc/logstash/logstash.conf
远程11，返回启动的logstash主机，查看数据
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
