ELK日志平台是一个完整的日志分析系统,有三个开源工具构建组成,分别是:Elasticsearch、Logstash和Kibana。Elasticsearch用于数据分析和深度搜索;Logstash作用是从其他服务器上传输和转发日志,对其集中管理,进行分析;Kibana则是提供了强大的UI展示,将数据可视化。
安装ELK日志平台
ELK基础环境需要java环境,官网要求5.x版本要大于java8。而且安装方式多样化,支持zip、tar.gz、rpm包、deb包、window环境还有docker环境。根据自己喜好选择吧。
我选择的是yum安装,简单方便,系统要求的话没有辣么严格,官网说yum安装方式不再支持centos5.x系列了,非要用centos5.x就去使用tar.gz包吧,官网有具体方法,不再复述。yum安装方式centos6.x和centos7.x都可以,但是我推荐用centos7.x安装,不知道为啥,感觉centos7.x支持更好,centos6.x装完经常会出问题。
还有一点需要说下就是,ELK各个组件版本要一致,官网要求的!
在一个就是安装顺序,为的是确保每个组件相互调用时都能正常运行:
1、Elasticsearch
• X-Pack for Elasticsearch
Kibana
• X-Pack for Kibana
LogstashBeatsElasticsearch Hadoop
安装Elasticsearch
1、导入Elasticsearch安装包PGP Key
rpm --import
https://artifacts.elastic.co/GPG-KEY-elasticsearch
2、创建yum源
[root@localhost ~]# cat >> /etc/yum.repos.d/elasticsearch.repo <<EOF
> [elasticsearch-5.x]
> name=Elasticsearch repository for 5.x packages
> baseurl=https://artifacts.elastic.co/packages/5.x/yum
> gpgcheck=1
> gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
> enabled=1
> autorefresh=1
> type=rpm-md
> EOF
3、安装、启动Elasticsearch进程并开机启动
[wangpeng@localhost ~]$ sudo yum install elasticsearch
[wangpeng@localhost ~]$ sudo /bin/systemctl daemon-reload
[wangpeng@localhost ~]$ sudo /bin/systemctl enable elasticsearch.service
[wangpeng@localhost ~]$ sudo systemctl start elasticsearch.service
4、检查Elasticsearch是否已经启动
查看9200、9300是否已经启动
[wangpeng@localhost ~]$ curl http://localhost:9200
{
"name" : "F5Mw8Pp",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "zVEeXtPNTaeH-TKah7Buzw",
"version" : {
"number" : "5.4.0",
"build_hash" : "780f8c4",
"build_date" : "2017-04-28T17:43:27.229Z",
"build_snapshot" : false,
"lucene_version" : "6.5.0"
},
"tagline" : "You Know, for Search"
}
5、配置Elasticsearch
rpm包配置文件在/etc/elasticsearch下面的elasticsearch.yml
vim /etc/elasticsearch/elasticsearch.yml
cluster.name: elasticsearch-test
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
[wangpeng@localhost ~]$ sudo systemctl restart elasticsearch.service
[wangpeng@localhost ~]$ curl http://localhost:9200
{
"name" : "node-1",
"cluster_name" : "elasticsearch-test",
"cluster_uuid" : "zVEeXtPNTaeH-TKah7Buzw",
"version" : {
"number" : "5.4.0",
"build_hash" : "780f8c4",
"build_date" : "2017-04-28T17:43:27.229Z",
"build_snapshot" : false,
"lucene_version" : "6.5.0"
},
"tagline" : "You Know, for Search"
}
6、将/etc/elasticsearch/配置拷贝到/usr/share/elasticsearch/config下面
[wangpeng@localhost ~]$ sudo mkdir /usr/share/elasticsearch/config
[wangpeng@localhost ~]$ sudo ln -sf /etc/elasticsearch/* /usr/share/elasticsearch/config/
[wangpeng@localhost ~]$ sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
[wangpeng@localhost ~]$ sudo systemctl restart elasticsearch.service
注意:这一点好多人不会注意,因为你不修复也不会启动失败,但是就是写不进数据进去,这个坑好久才发现,看下日志会报错,但是却能启动,我也是服了!~
7、装个head插件
这个插件5.X官网不再支持了,插件命令没有了,因为它有自己x-pack插件了,但是我装了x-pack发现着实让人吐血,有安全认证方面的问题,导致elk各种问题出现,目前还没研究明白,时间不充裕。
这个head插件我是直接抄的网上大神制作,略有改动。
7.1、下载并配置nodejs
由于head插件本质上还是一个nodejs的工程,因此需要安装node,使用npm来安装依赖的包。(npm可以理解为maven)
去官网下载nodejs,https://nodejs.org/en/download/
wget https://nodejs.org/dist/v8.1.1/node-v8.1.1-linux-x64.tar.xz
tar xf node-v8.1.1-linux-x64.tar.xz
mv node-v8.1.1-linux-x64 /usr/local/node
chown -R elasticsearch:elasticsearch /usr/local/node
ln -sf /usr/local/node/bin/node /usr/bin/node
ln -sf /usr/local/node/bin/npm /usr/bin/npm
7.2、安装grunt
npm install -g grunt-cli
ln -sf /usr/local/node/bin/grunt /usr/bin/grunt cd /var/lib/elasticsearch
7.3、下载、安装并配置head
yum -y install git
cd /var/lib/elasticsearch
git clone git://github.com/mobz/elasticsearch-head.git
chown -R elasticsearch:elasticsearch elasticsearch-head/
cd elasticsearch-head/
npm install
7.4、配置head文件
[root@elk-node1 ~]# cd /var/lib/elasticsearch/elasticsearch-head/
vim Gruntfile.js
connect: {
server: {
options: {
port: 9100,
hostname: "0.0.0.0",
base: '.',
keepalive: true
}
}
}
[root@elk-node1 elasticsearch-head]# cd _site/
[root@elk-node1 _site]# vim app.js
把localhost修改成你es的服务器地址:
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://10.10.10.10:9200";
7.5、启动head插件
grunt server &
安装Kibana
1、yum安装Kibana
[wangpeng@localhost ~]$ sudo yum install kibana
rpm包配置文件在/etc/kibana下面的kibana.yml
/etc/kibana/kibana.yml
2、配置Kibana文件
[wangpeng@localhost ~]$ vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
3、启动并设置开机启动
[wangpeng@localhost ~]$ sudo systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service
[wangpeng@localhost ~]$ sudo systemctl start kibana.service
安装Logstash
1、yum安装Logstash
[wangpeng@localhost ~]$ sudo yum -y install logstash
[wangpeng@localhost ~]$ sudo systemctl start logstash.service
[wangpeng@localhost ~]$ sudo ln -s /usr/share/logstash/bin/logstash /usr/bin/logstash
[wangpeng@localhost ~]$ sudo -u logstash sh -c 'mkdir -pv /usr/share/logstash/config'
[wangpeng@localhost ~]$ sudo -u logstash sh -c 'ln -s /etc/logstash/* /usr/share/logstash/config/'
2、测试Logstash是否能正常运行
[wangpeng@localhost ~]$ sudo logstash -e 'input {stdin{}}output { stdout{}}'
hello world
2017-06-02T07:14:13.130Z localhost hello world
[wangpeng@localhost ~]$ sudo logstash -e 'input {stdin{}}output {
stdout{codec=>rubydebug}}'
hello world
The stdin plugin is now waiting for input:
{
"@timestamp" => 2017-06-02T07:17:44.053Z,
"@version" => "1",
"host" => "localhost",
"message" => "hello world"
}
3、写个测试文件,测试一下es是否能够接受数据
[wangpeng@localhost ~]$ vim /etc/logstash/conf.d/test.conf
input{
stdin{}
}
output{
elasticsearch{
hosts => "127.0.0.1:9200"
index => "test-messages-%{+YYYY.MM.dd}"
}
}
[wangpeng@localhost ~]$ logstash -f /etc/logstash/conf.d/test.conf -t
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
[wangpeng@localhost ~]$ logstash -f /etc/logstash/conf.d/test.conf
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
The stdin plugin is now waiting for input:
hello world
this is test message
study logstash
4、Kibana里添加该索引(测试),只要es里面能产生索引,Kibana就能加在上去
安装调试先到这里。
https://blog.51cto.com/wangpengtai/1939138