elk 设置索引模版 声明周期策略 elk 索引更新

前言：当ELK平台运行的时间越来越长，对服务器磁盘占用空间就会越来越大，传统的清理index索引数据是脚本+计划任务，虽然能够解决定时清理数据的需求，当索引越来越多，每次都要去修改脚本并不方面。ELK官方在6.6版本推出了 ILM (index lifecycle Management) 满足以上的需求

当前的ELK 版本为7.9.3

filebeat.yml文件

filebeat.inputs:
- type: filestream
  enabled: true
  id: id9
  tags: ["test-yingjian"]
  paths:
    - /root/access.log
  fields:
    log_topics: test-yingjian

  # 这个是用来处理异常产生多行数据时，将多行数据当作一条日志处理，根据自己的异常日志的格式做修改
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  tail_files: true
  ignore_older: 1h

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  
  # Set to true to enable config reloading
  reload.enabled: false

output.logstash:
  hosts: ["172.18.138.10:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

logstash.conf 文件

input {
    beats {
	port => 5044
        client_inactivity_timeout => 36000
    }

filter {
    mutate {
        
    }
}

output {
    if "system" in [type]  {
        elasticsearch {
            hosts => ["http://172.18.138.10:9200"]
            # 索引名称，没有会自动创建
            index => "%{tags}-%{+YYYY.MM}"
            # 账号
            user => "elastic"
            # 密码
            password => "********"
        }
    } else if "test-yingjian" in [tags] {
         elasticsearch {
            hosts => ["http://172.18.138.10:9200"]
            # 索引名称，没有会自动创建
            index => "test" #对应创建的索引别名
            # 账号
            user => "elastic"
            # 密码
            password => "********"
        }
    } else {
    	elasticsearch {
            hosts => ["http://172.18.138.10:9200"]
            # 索引名称，没有会自动创建
            index => "%{[fields][log_topics]}"
            # 账号
            user => "elastic"
            # 密码
            password => "********"
        }
    }	
    file {
     #   path => "/data/eslog/%{+YYYY-MM-dd}/%{tags}.log"
        path => "/data/eslog/%{+YYYY-MM-dd}/%{[fields][log_topics]}.log"
    }
}

elasticsearch.yml

cluster.name: es_junzun_pro  #集群名称
node.master: true #主节点 主要用于元数据(metadata)的处理，比如索引的新增、删除、分片分配等
node.data: true  #数据节点 这样的节点负责存储数据。后期提供存储和查询服务
node.attr.rack: r1  # hot节点标识
node.attr.hotwarm_type: hot    #节点类型 [hotwarm] 自定义
node.name: es_jinzun
path.data: /data/elastic/data  #es 存储目录
path.logs: /data/elastic/logs  #es 日志目录
bootstrap.memory_lock: true    #不使用交换分区
network.host: 0.0.0.0          
http.port: 9200
discovery.seed_hosts: ["172.18.138.10"]
transport.tcp.port: 9300
cluster.initial_master_nodes: ["es_jinzun"]
xpack.security.enabled: true  #开启X-pack 认证
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

# x-head 访问配置  master节点配置
http.cors.enabled: true
http.cors.allow-origin: "*"

配置ilm生命管理周期

脚本创建 ilm 策略

在kibana开发工具操作

PUT _ilm/policy/test_policy
{
  "policy": {                       
    "phases": {
      "hot": {                      
        "actions": {
          "rollover": {             
            "max_size": "50GB",
            "max_age": "30d",
            "max_docs" : 10
          }
        }
      },
      "delete": {
        "min_age": "30s",           
        "actions": {
          "delete": {}              
        }
      }
    }
  }
}

配置索引模板绑定ilm策略

脚本创建索引模板

PUT _template/test_template
{
  "index_patterns" : ["test-*"],
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 0,
    "index.lifecycle.name": "test_policy",
    "index.routing.allocation.require.hotwarm_type":"hot",
    "index.lifecycle.rollover_alias": "test"
  }
}

基于序号创建初始索引

PUT test-000001
{
"aliases": {
    "test":{   #test 索引别名 对应 rollover_alias 配置
       "is_write_index": true  #该索引可以被写入
        }
      }
}

配置建议

1. logstash 基于x-pack 传输数据到es 该账号我设置是elastic用户 生产环境建议单独创建一个用户用于传输数据
2. logstash 对输出到es中的日志数据在本地备份，当日志数据需要重新导入的时候 不需要在每一台服务器上操作，只需要在logstash配置重新采集即可
3. es 集群master节点保证可用性 3台以上并且为奇数
4. es 集群node节点 作为数据节点 对应cpu 内存 硬盘 要求比较高
5. es 每个索引的分片大小控制在 30GB-50GB大小
6. es 主分片+副本分片数量 总和为1.5~3倍数量的node节点数量为适
7. es refresh_interval flush 磁盘时间间隔 默认为1s， 根据不同的日志调整该刷新间隔时间，时间越短对服务器压力就越大 （即同一时刻从内存写入到磁盘的日志就越多）
8. logstash 配置文件中 移除掉不需要的字段

测试filebeat写入数据

当前手动写入9条数据

再写入一条数据，查看索引是否会自动轮转后，并且30s后删除

文档所在的索引为

由于我操作慢了点30s后 test-000001已经被自动删除了

现在再去查询数据，会出现找不到数据

重新写入数据到es，此时数据存在的索引为test-000002

查看冷热节点状态

GET _cat/nodeattrs?v&h=host,attr,value

查看logstash本地备份数据

{"ecs":{"version":"1.12.0"},"@version":"1","agent":{"id":"c0b8166d-df20-4348-aa57-79c08ad2e552","ephemeral_id":"5e35edad-cb7d-48df-aa15-a4dafca785bc","name":"guojing","hostname":"guojing","version":"7.17.4","type":"filebeat"},"input":{"type":"filestream"},"fields":{"log_topics":"test-yingjian"},"@timestamp":"2022-07-28T07:35:59.500Z","log":{"file":{"path":"/root/access.log"},"offset":2095},"tags":["test-yingjian","beats_input_codec_plain_applied"],"host":{"name":"guojing"},"message":"10"}
{"@version":"1","ecs":{"version":"1.12.0"},"agent":{"id":"c0b8166d-df20-4348-aa57-79c08ad2e552","name":"guojing","ephemeral_id":"5e35edad-cb7d-48df-aa15-a4dafca785bc","hostname":"guojing","version":"7.17.4","type":"filebeat"},"input":{"type":"filestream"},"fields":{"log_topics":"test-yingjian"},"@timestamp":"2022-07-28T07:40:23.513Z","log":{"file":{"path":"/root/access.log"},"offset":2098},"tags":["test-yingjian","beats_input_codec_plain_applied"],"host":{"name":"guojing"},"message":"1"}
{"@version":"1","ecs":{"version":"1.12.0"},"agent":{"id":"c0b8166d-df20-4348-aa57-79c08ad2e552","ephemeral_id":"5e35edad-cb7d-48df-aa15-a4dafca785bc","name":"guojing","hostname":"guojing","version":"7.17.4","type":"filebeat"},"input":{"type":"filestream"},"fields":{"log_topics":"test-yingjian"},"@timestamp":"2022-07-28T07:40:23.513Z","log":{"file":{"path":"/root/access.log"},"offset":2100},"tags":["test-yingjian","beats_input_codec_plain_applied"],"host":{"name":"guojing"},"message":"2"}
{"ecs":{"version":"1.12.0"},"@version":"1","agent":{"id":"c0b8166d-df20-4348-aa57-79c08ad2e552","name":"guojing","ephemeral_id":"5e35edad-cb7d-48df-aa15-a4dafca785bc","hostname":"guojing","version":"7.17.4","type":"filebeat"},"input":{"type":"filestream"},"fields":{"log_topics":"test-yingjian"},"@timestamp":"2022-07-28T07:40:23.513Z","log":{"file":{"path":"/root/access.log"},"offset":2102},"tags":["test-yingjian","beats_input_codec_plain_applied"],"host":{"name":"guojing"},"message":"3"}
{"@version":"1","ecs":{"version":"1.12.0"},"agent":{"id":"c0b8166d-df20-4348-aa57-79c08ad2e552","name":"guojing","ephemeral_id":"5e35edad-cb7d-48df-aa15-a4dafca785bc","hostname":"guojing","version":"7.17.4","type":"filebeat"},"input":{"type":"filestream"},"fields":{"log_topics":"test-yingjian"},"@timestamp":"2022-07-28T07:40:23.513Z","log":{"offset":2104,"file":{"path":"/root/access.log"}},"tags":["test-yingjian","beats_input_codec_plain_applied"],"host":{"name":"guojing"},"message":"4"}
{"@version":"1","ecs":{"version":"1.12.0"},"agent":{"id":"c0b8166d-df20-4348-aa57-79c08ad2e552","ephemeral_id":"5e35edad-cb7d-48df-aa15-a4dafca785bc","name":"guojing","hostname":"guojing","version":"7.17.4","type":"filebeat"},"input":{"type":"filestream"},"fields":{"log_topics":"test-yingjian"},"@timestamp":"2022-07-28T07:40:25.514Z","log":{"file":{"path":"/root/access.log"},"offset":2106},"tags":["test-yingjian","beats_input_codec_plain_applied"],"host":{"name":"guojing"},"message":"5"}

当es索引删除时可用该数据还原

服务器定时清理logstash es 日志数据

#!/usr/bin/bash

#Tate:2022.4.26
#Author:Yingjian
#function: 定时清理elasticsearch、logstash日志文件
#env

workdir=`cd $(dirname $0);pwd`
seven_day_ago=`date -d '-7 day' "+%F"`

cat > $workdir/es_log_dir.txt << EOF
/data/elastic/logs
/usr/local/services/logstash/logs
EOF

while read line
do
find $line -type f -mtime +7 -exec rm -f {} \; 
done < $workdir/es_log_dir.txt

rm -rf /data/eslog/$seven_day_ago