​

​​https://www.jianshu.com/p/d3042a08eb5e​​

​


负责发送日志的所有服务器安装logstash

1.下载安装包

wget https://download.elastic.co/logstash/logstash/logstash-2.4.1.tar.gz



2.安装

tar zxvf logstash-2.4.1.tar.gz

mv logstash-2.4.1 /usr/local/logstash

mkdir -p /usr/local/logstash/etc



3.修改配置文件

vim /usr/local/logstash/etc/logstash.cnf

添加

input {
    file {
        type => "tomcat-catalina"
        path => "/u02/8080-tomcat/logs/catalina.out"
        codec => multiline {
            pattern => "^\s"
            what => "previous"
        }
    }
}


output {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-tomcat-catalina"
    }
}





4.如果系统中没设置环境变量，需要添加JAVA_HOME环境变量

vim /usr/local/logstash/bin/logstash.lib.sh

添加

JAVA_HOME=/usr/java/jdk1.7.0_79


5.启动

mkdir -p /usr/local/logstash/logs

nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/etc/logstash.cnf >> /usr/local/logstash/logs/nohup.out 2>&1 &


6.到redis查看

redis-cli

LPOP "logstash-tomcat-catalina"






负责接收数据的logstash服务器安需要安装logstash，并使用以下配置文件

vim /usr/local/logstash/etc/logstash.cnf

添加

input {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-tomcat-catalina"
    }
}


filter {
    ruby {
        code => "event['filedatetag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
    }

}

output {
    file {
         path => "/data/log/tomcat/%{host}/catalina-%{filedatetag}.log"
         message_format=>"%{host}----%{message}"
    }
    #stdout{
    #    codec=>rubydebug
    #}

    if [message] =~ "Exception" {
        file {
            path => "/data/log/tomcat/exception/exception-%{filedatetag}.log"
        }
    }
}


收集nginx日志的配置文件

客户端上的配置

vim /usr/local/logstash/etc/nginx_log.cnf


input {
    file {
        type => "nginx_access_log"
        path => "/data/logs/nginx/access_log.log"
    }
    file {
        type => "nginx_access"
        path => "/data/logs/nginx/access.log"
    }
    file {
        type => "nginx_access_check"
        path => "/data/logs/nginx/access_check.log"
    }
}


output {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-nginx-log"
    }
}



日志服务器上的配置

vim /usr/local/logstash/etc/nginx_log.cnf


input {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-nginx-log"
    }
}


filter {
    ruby {
        code => "event['filedatetag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
    }

}

output {
    file {
         path => "/data/log/nginx/%{host}/%{type}/%{type}-%{filedatetag}.log"
         message_format=>"%{message}"
    }
}
input {
    redis {
        host => "172.17.17.15"
        port => 6379
        data_type => "list"
        key => "logstash-nginx-log"
    }
}


filter {
    ruby {
        code => "event['filedatetag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
    }
    json {
        source => "message"
        target => "jsoncontent"
    }

}

output {
    file {
         path => "/data/log/nginx/all/%{type}/%{type}-%{filedatetag}.log"
         message_format=>"%{message}"
         #message_format=>"%{host}----%{type}----%{message}"
    }


    if [message] =~ "code=514" {
        file {
            path => "/data/log/nginx/zabbix_monitor/yunxin-code514.log"
            message_format=>"%{message}"
        }
    }


    exec {
        command => "/usr/local/redis/bin/redis-cli -h 127.0.0.1 incr zabbix_nginx_log_count_%{type}"
    }
    exec {
        command => "/usr/local/redis/bin/redis-cli -h 127.0.0.1 incr zabbix_nginx_log_count_%{type}_%{[jsoncontent][status]}"
    }
    #stdout{
    #    codec=>rubydebug
    #}
}