Logstash 入门实战(2)--安装及使用

原创

且行且码 2022-10-03 15:04:14 ©著作权

文章标签 Elastic safari chrome nginx 文章分类 虚拟化云计算

©著作权归作者所有：来自51CTO博客作者且行且码的原创作品，请联系作者获取转载授权，否则将追究法律责任

本文主要介绍 Logstash 的安装及简单的使用，相关的环境及软件信息如下：CentOS 7.9、Logstash 8.2.2、Elasticsearch 8.2.2。

1、安装

根据环境下载对应的安装包：https://www.elastic.co/cn/downloads/logstash，这里选择 Linux x86_64 版本；下载完成后在服务器上解压即可：

tar zxvf logstash-8.2.2-linux-x86_64.tar.gz

2、简单使用

这里使用 Logstash 来收集 Nginx 访问日志并保存到 Elasticsearch 中。

2.1、创建索引

先创建用于存放 Nginx 访问日志的索引。

curl -X PUT -H 'Content-Type:application/json' 'http://10.49.196.11:9200/nginx-index' -d '
{
  "mappings": {
    "properties": {
      "message": {
        "type": "text": {
        "type": "text": {
        "type": "text": {
        "type": "date": {
        "type": "keyword": {
        "type": "text": {
        "type": "keyword": {
        "type": "keyword": {
        "type": "integer": {
        "type": "integer": {
        "type": "text": {
        "type": "text"
      }
    }
  }
}'

2.2、Logstash 配置输入

input {
  file {
    path => ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
    start_position => "beginning"
  }
}

这里指定了 Nginx 日志文件的路径。

2.3、Logstash 配置过滤器

我们需要对日志进行处理，提取出我们需要的字段。

filter {
  grok {
    match => { "message" => "%{IP:ip} - %{USER:remoteUser} \[%{HTTPDATE:accessTimeStr}\] \"%{WORD:method} %{URIPATHPARAM:path} %{WORD:
protocal}/%{NUMBER:version}\" %{INT:status} %{INT:bytes} \"%{DATA:referer}\" \"%{DATA:userAgent}\"" }
  }

  if [tags][0] == '_grokparsefailure' {
    drop{}
  }
  
  date {
    match => ["accessTimeStr", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "accessTime"
  }
  
  mutate {
    convert => {
      "bytes" => "integer"
      "status" => "integer"
    }
  }
  
  prune {
    blacklist_names => ["log","@version","host","@timestamp","accessTimeStr","event"]
  }
}

grok 插件通过正则表达把原始日志拆分成相应的字段；date 插件把字段转成日期格式；mutate 插件把字段转成我们需要的类型；prune 插件过滤出不需要存到 Elasticsearch 的字段。

2.4、Logstash 配置输出

配置输出到本地的 Elasticsearch。

output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-index"
  }
}

2.5、完整配置

Logstash 入门实战(2)--安装及使用_chrome

Logstash 入门实战(2)--安装及使用_Elastic_02

input {
  file {
    path => ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{IP:ip} - %{USER:remoteUser} \[%{HTTPDATE:accessTimeStr}\] \"%{WORD:method} %{URIPATHPARAM:path} %{WORD:
protocal}/%{NUMBER:version}\" %{INT:status} %{INT:bytes} \"%{DATA:referer}\" \"%{DATA:userAgent}\"" }
  }

  if [tags][0] == '_grokparsefailure' {
    drop{}
  }
  
  date {
    match => ["accessTimeStr", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "accessTime"
  }
  
  mutate {
    convert => {
      "bytes" => "integer"
      "status" => "integer"
    }
  }
  
  prune {
    blacklist_names => ["log","@version","host","@timestamp","accessTimeStr","event"]
  }
}


output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-index"
  }
}

nginx.conf

2.6、运行 Logstash

bin/logstash -f nginx.conf

2.7、验证

Nginx 的访问日志信息如下：

10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
...

Logstash 的控制台日志信息如下：

{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:15Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:16Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:16Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
...

Elasticsearch 中查询数据：

curl -X GET -H 'Content-Type:application/json' 'http://10.49.196.11:9200/nginx-index'

结果如下：

{
  "took": 530,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "nginx-index",
        "_id": "nSjnFYMB-RPngHUTzpDo",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:15Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      },
      {
        "_index": "nginx-index",
        "_id": "nijnFYMB-RPngHUT0JCK",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:16Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      },
      {
        "_index": "nginx-index",
        "_id": "nyjnFYMB-RPngHUT0JCK",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:16Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      }
    ]
  }
}