CentOS 6.5 x64安装ELK日志分析系统

环境说明:

系统是CentOS 6.5x64,已经安装好编译环境

已经安装好Tengine2.10

已经关闭iptables

核心软件包如下:

elasticsearch-1.4.2.tar.gz

下载链接:

https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz

logstash-1.4.2.tar.gz

下载链接:

https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

kibana-3.1.2.tar.gz

下载链接:

https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz

特别声明一点elasticsearchkibana有版本兼容问题,很多人,网页死都出不来,就是因为这个原因。

所以,如果要参考这篇文章,请严格按照版本来,不要下载最新版。

 

安装JDK

ElasticSearchLogstash依赖于JDK

tar zxvfjdk-7u71-linux-x64.tar.gz

mv jdk1.7.0_71/usr/local/java

vi /etc/profile

最后一行添加

#Java environment

exportJAVA_HOME=/usr/local/java

exportPATH=$PATH:$JAVA_HOME/bin

exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib:$JAVA_HOME/bin

加载环境变量

source /etc/profile

查看版本

[root@localhost ~]#java -version

java version"1.7.0_45"

OpenJDK RuntimeEnvironment (rhel-2.4.3.3.el6-x86_64 u45-b15)

OpenJDK 64-Bit ServerVM (build 24.45-b08, mixed mode)

安装ElasticSearch

ElasticSearch默认的对外服务的HTTP端口是9200,节点间交互的TCP端口是9300


tar zxvfelasticsearch-1.4.2.tar.gz

mv elasticsearch-1.4.2/usr/local/elasticsearch

 

修改ElasticSearch的配置文件,追加一行内容,否则Kibana页面提示Connection Failed


echo"http.cors.enabled: true" >>/usr/local/elasticsearch/config/elasticsearch.yml

 

安装elasticsearch-servicewrapper

wgethttps://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.tar.gz

tar zxvf master.tar.gz

cp -relasticsearch-servicewrapper-master/service /usr/local/elasticsearch/bin/

启动ElasticSearch服务

/usr/local/elasticsearch/bin/service/elasticsearchstart

等待5秒,查看端口

[root@localhost jar]# netstat -anpt | grep 9200

tcp       0      0 :::9200                     :::*                        LISTEN      15682/java

 

测试ElasticSearch服务是否正常,预期返回200的状态码:

 

[root@localhost ~]# curl -X GET http://localhost:9200

{

  "status": 200,

  "name" : "Whistler",

  "cluster_name" :"elasticsearch",

  "version" : {

    "number" : "1.4.2",

    "build_hash" :"927caff6f05403e936c20bf4529f144f0c89fd8c",

    "build_timestamp" :"2014-12-16T14:11:12Z",

    "build_snapshot" : false,

    "lucene_version" :"4.10.2"

  },

  "tagline" : "You Know, forSearch"

}

 

安装Logstash

Logstash默认的对外服务的端口是9292

tar zxvflogstash-1.4.2.tar.gz

mv logstash-1.4.2/usr/local/logstash

mkdir -p/usr/local/logstash/etc

创建Logstash配置文件logstash_agent.conf,这里将Nginx日志和messages系统日志作为输入,输出直接传给ElasticSearch

里面的IP地址是服务器的IP地址

vim /usr/local/logstash/etc/logstash_agent.conf

 

input {

  file {

    type => "nginx.access"

    path =>["/usr/local/nginx/logs/access.log"]

  }

 

  file {

    type => "nginx.error"

    path =>["/usr/local/nginx/logs/error.log"]

  }

 

output {

  elasticsearch {

    host => "192.168.1.105"

    port => 9300

  }

}

启动logstash

/usr/local/logstash/bin/logstash-f /usr/local/logstash/etc/logstash_agent.conf &

 

编辑index配置文件,主要负责解析日志格式的

vi logstash_indexer.conf

filter部分删除,添加新的filter。效果如下:

 

input {

  file {

    type => "nginx.access"

    path => ["/usr/local/nginx/logs/access.log"]

  }

}


filter {

        grok {

                type => "nginx.access"

                match => [

"message", "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}"

                ]

        }

}



output {

        elasticsearch {

                embedded => false

                protocol => "http"

                host => "192.168.1.105"

                port => "9200"

        }

}

 

启动index配置文件

/usr/local/logstash/bin/logstash-f /usr/local/logstash/etc/logstash_indexer.conf &

 

安装Kibana

Kibana的源代码就是一个网页,所以直接用Nginx发布就可以了。

mkdir /www

tar zxvfkibana-3.1.2.tar.gz

mv kibana-3.1.2/www/kibana

修改Kibana的配置文件,指定elasticsearch

vim /www/kibana/config.js

 

32     elasticsearch:"http://"+window.location.hostname+":9200",

修改为:

elasticsearch:"http://192.168.1.105:9200",

 

修改Nginx配置文件

vim/usr/local/nginx/conf/vhosts/kibana.conf


server {

        listen   80;

        server_name kibana.xx.com;

        root    /www/kibana;

        index index.php index.html index.htm;



        location / {


        }


        location ~ \.php$ {

                fastcgi_pass   127.0.0.1:9000;

                fastcgi_index  index.php;

                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

                include        fastcgi_params;

        }


}


重新加载nginx配置

/usr/local/nginx/sbin/nginx-s reload

访问页面

http://kibana.xx.com/

wKioL1YLlRXyib3AAAHkAnsDb14631.jpg

提示LogStash仪表板设置

点击右边的样品仪表板

wKioL1YLlTewecBUAAE9aY0vSvM248.jpg

进入页面

wKioL1YLlWTj8GB6AAGtGN-bdsE753.jpg

在下面就可以看到Nginx日志了

wKiom1YLlZiC7engAAK8am25dFQ028.jpg

点击source_ip,选择bar

wKiom1YLlc_SGVaVAAHaInhlwZo102.jpg

可以看到排名前十的客户端访问IP地址

wKioL1YLlgqQs4JCAAE9fsOMnbM722.jpg

其他的功能可以自己慢慢看


本文参考

http://blog.csdn.net/i_chips/article/details/43309415