Elasticsearch + logstash + kibana 配置

原创

mb60ffb84ea5705 2021-07-27 16:29:09 博主文章分类：ELK ©著作权

文章标签 编程 文章分类 Java 后端开发

©著作权归作者所有：来自51CTO博客作者mb60ffb84ea5705的原创作品，请联系作者获取转载授权，否则将追究法律责任

Elasticsearch 配置

Elasticsearch不仅仅是Lucene和全文搜索，我们还能这样去描述它：

分布式的实时文件存储，每个字段都被索引并可被搜索
分布式的实时分析搜索引擎
可以扩展到上百台服务器，处理PB级结构化或非结构化数据

Elastic Search

索引基本单位是文档文档类型， ES 分片横向扩展。分片可以做备份，节点，一个es的实例就是一个节点（集群使用）

git clone https://github.com/elasticsearch/elasticsearch-servicewrapper.git 管理脚本

Elasticsearch ⇒ 索引 ⇒ 类型 ⇒ 文档 ⇒ 字段(Fields)

本文基于elasticsearch最新版搭建的下载地址.

开发环境：centos7 ：

192.168.20.153 zookeeper1
192.168.20.154 zookeeper2
192.168.20.155 zookeeper3
192.168.20.206 kafka1
192.168.20.207 kafka2
192.168.20.208 kafka3
192.168.20.204 logstashserver
192.168.20.205 kibana
192.168.20.201 es1
192.168.20.202 es2
192.168.20.203 es3

路径位置：

/data/elasticsearch

/data/java/

elasticsearch的集群搭建相当简单，不像solrcloud的搭建那么复杂（需要自己安装zookeeper），

在es1上操作：

下载elasticsearch：

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.5/elasticsearch-2.3.5.tar.gz

解压：

tar -zxvf elasticsearch-2.3.5.tar.gz

mv elasticsearch-2.3.5 elasticsearch

接下来创建el用户，因为elasticsearch不允许以root运行（其实也可以运行，需要配置）。

设置虚拟 –Xms -Xmx 内存大小

vim /data/elasticsearch/bin/elasticsearch.in.sh

设置虚拟内存

echo “vm.max_map_count=262144” >> /etc/sysctl.conf

关闭swap分区

Swapoff –a

修改配置文件fstab

#/dev/mapper/centos-swap swap swap defaults 0 0

设置mlocall 为true

vim /data/elasticsearch/config/elasticsearch.yml

bootstrap.memory_lock: true

5.设置elastrisearch用户名密码

useradd el

passwd el

123456

chown -R el:el elasticsearch

su el

cd elasticsearch/conf/

修改配置文件：

vim elasticsearch.yml

cluster.name: feng

node.name: es1

network.host: 192.168.20.201

#####以下配置是防止脑裂###########

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping_timeout:120s

client.transport.ping_timeout: 60s

discovery.zen.ping.unicast.hosts: [“es1”, “es2″,”es3”]

配置信息的具体讲解先放在后边：直接按上边配置即可

启动

su - el

/data/elasticsearch/bin/elasticsearch -d

安装Marvel（）

Marvel是Elasticsearch的管理和监控工具，在开发环境下免费使用。它包含了一个叫做Sense的交互式控制台，使用户方便的通过浏览器直接与Elasticsearch进行交互。

bin/plugin install license

bin/plugin install marvel-agent

接下来，我们安装head插件：

cd elasticsearch/bin/

通过如下命令安装head插件

./plugin install mobz/elasticsearch-head

如果下载失败，可以自己手动下载，下载地址：

并上传到elasticsearch所在目录：

使用如下方式进行安装：

./plugin install file:/java/elasticsearch-head-master.zip

安装中文分词插件：ik

ik是在github上，需要自己下载并使用maven编译：github地址如下：

git clone https://github.com/medcl/elasticsearch-analysis-ik
cd elasticsearch-analysis-ik
mvn clean
mvn compile
mvn package

需要对比elasticsearch 与 ik 的对应关系

Elasticsearch2.4 与 ik没有找到对应关系， Elasticsearch5.00 的head的插件不支持，所以使用2.3.5

curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.5/elasticsearch-2.3.5.tar.gz

cd elasticsearch-analysis-ik

git tag -l 显示标记

git checkout –b dev1.9.5 v1.9.5 使用1.9.5版本

mvn clean

mvn compile

mvn package

mkdir /data/elasticsearch/plugins/ik -p

unzip /data/tools/elasticsearch-analysis-ik/target/releases/elasticsearch-analysis-ik-1.9.5.zip -d /data/elasticsearch/plugins/ik

su root

给其他两台机器拷贝：

scp -r elasticsearch es2:/data/

scp -r elasticsearch es3:/data/

其他两台机器同样也需要创建新的用户el，并赋予相应的权限，这里不再赘述。

登录es2：

vim elasticsearch/config/elasticsearch.yml

node.name: es2

network.host: 192.168.20.202

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping_timeout:120s

client.transport.ping_timeout: 60s

discovery.zen.ping.unicast.hosts: [“es1”, “es2″,”es3”]

登录es3：

vim elasticsearch/config/elasticsearch.yml

node.name: es3

network.host: 192.168.20.203

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping_timeout:120s

client.transport.ping_timeout: 60s

discovery.zen.ping.unicast.hosts: [“es1”, “es2″,”es3”]

接下来，启动elasticsearch cluster

分别在es1，es2，es3上执行如下操作：

su el

./elasticsearch/bin/elasticsearch -d

到这里，集群已经成功启动：

通过如下地址可验证集群是否启动成功：

http://es1:9200/_plugin/head/

出现如下图所示，则说明，集群启动成功。

接下来，让我们通过如下方式建立并索引文档：

1.create a index

curl -XPUT http://localhost:9200/index

2.create a mapping

curl -XPOST http://localhost:9200/index/fulltext/_mapping -d'

{

"fulltext": {

"_all": {

"analyzer": "ik_max_word",

"search_analyzer": "ik_max_word",

"term_vector": "no",

"store": "false"

"properties": {

"content": {

"type": "text",

"analyzer": "ik_max_word",

"search_analyzer": "ik_max_word",

"include_in_all": "true",

"boost": 8

}

3.index some docs

curl -XPOST http://localhost:9200/index/fulltext/1 -d'

{"content":"美国留给伊拉克的是个烂摊子吗"}

curl -XPOST http://localhost:9200/index/fulltext/2 -d'

{"content":"公安部：各地校车将享最高路权"}

curl -XPOST http://localhost:9200/index/fulltext/3 -d'

{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}

curl -XPOST http://localhost:9200/index/fulltext/4 -d'

{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击嫌犯已自首"}

4.query with highlighting

curl -XPOST http://localhost:9200/index/fulltext/_search -d'

{

"query" : { "match" : { "content" : "中国" }},

"highlight" : {

"pre_tags" : ["<tag1>", "<tag2>"],

"post_tags" : ["</tag1>", "</tag2>"],

"fields" : {

"content" : {}

}

Result

{

"took": 14,

"timed_out": false,

"_shards": {

"total": 5,

"successful": 5,

"failed": 0

"hits": {

"total": 2,

"max_score": 2,

"hits": [

{

"_index": "index",

"_type": "fulltext",

"_id": "4",

"_score": 2,

"_source": {

"content": "中国驻洛杉矶领事馆遭亚裔男子枪击嫌犯已自首"

"highlight": {

"content": [

"<tag1>中国</tag1>驻洛杉矶领事馆遭亚裔男子枪击嫌犯已自首 "

]

}

{

"_index": "index",

"_type": "fulltext",

"_id": "3",

"_score": 2,

"_source": {

"content": "中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"

"highlight": {

"content": [

"均每天扣1艘<tag1>中国</tag1>渔船 "

]

}

]

}

看到如下图说明我们集群搭建成功并可正常使用：

删除索引:

curl -XDELETE 'http://localhost:9200/feng'

获取数据

curl –XGET 'http://localhost:9200/index/fulltext/2'

/index/fulltext/2

索引/类型/id

删除数据

curl –XDELETE 'http://localhost:9200/index/fulltext/2'

例如：删除索引为

[root@es1 nodes]# curl -XDELETE 'http://192.168.20.201:9200/test-system-messages-2016-09'
{"acknowledged":true}

[root@es1 nodes]# curl -XDELETE 'http://192.168.20.201:9200/test-system-messages-2016.09.270'
{"acknowledged":true}

[root@es1 nodes]# curl -XDELETE 'http://192.168.20.201:9200/test-system-messages-2016.09.271

{"acknowledged":true}

logstash 客户端安装(安装在应用服务器上)

1. 下载

wget https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz

tar -zxvf logstash-2.4.0.tar.gz

mv logstash-2.4.0 /data/

ln -s logstash-2.4.0/ logstash

cd /data/logstash

mkdir logs etc

2 提供logstash管理脚本，其中里面的配置路径可根据实际情况修改

#!/bin/bash

FILE='/data/logstash/etc/*.conf' #logstash配置文件
LOGBIN='/data/logstash/bin/logstash agent --verbose --config' #指定logstash配置文件的命令
LOCK='/data/logstash/locks' #用锁文件配合服务启动与关闭
LOGLOG='--log /data/logstash/logs/stdou.log' #日志
START() {
if [ -f $LOCK ];then
echo -e "Logstash is already \033[32mrunning\033[0m, do nothing."
else
echo -e "Start logstash service.\033[32mdone\033[m"
nohup ${LOGBIN} ${FILE} ${LOGLOG} &
touch $LOCK
fi
}
STOP() {
if [ ! -f $LOCK ];then
echo -e "Logstash is already stop, do nothing."
else
echo -e "Stop logstash serivce \033[32mdone\033[m"
rm -rf $LOCK
ps -ef | greplogstash | grep -v "grep" | awk '{print $2}' | xargskill -s 9 >/dev/null
fi
}
STATUS() {
psaux | greplogstash | grep -v "grep" >/dev/null
if [ -f $LOCK ] && [ $? -eq 0 ]; then
echo -e "Logstash is: \033[32mrunning\033[0m..."
else
echo -e "Logstash is: \033[31mstopped\033[0m..."
fi
}
TEST(){
${LOGBIN} ${FILE} --configtest
}
case "$1" in
start)
START
;;
stop)
STOP
;;
status)
STATUS
;;
restart)
STOP
sleep 2
START
;;
test)
TEST
;;
*)
echo "Usage: /etc/init.d/logstash (test|start|stop|status|restart)"
;;
esac

3. logstash 把日志写入到kafka集群

cat /data/logstash/etc/logstash.conf

input {            #这里的输入还是定义的是从日志文件输入
  file {
    type => "system-message" 
    path => "/var/log/messages"
    start_position => "beginning"
  }
}
output {
    #stdout { codec => rubydebug }   #这是标准输出到终端，可以用于调试看有没有输出，注意输出的方向可以有多个
    kafka {  #输出到kafka
      bootstrap_servers => "192.168.2.22:9092,192.168.2.23:9092,192.168.2.24:9092"  #他们就是生产者
      topic_id => "system-messages"  #这个将作为主题的名称，将会自动创建
      compression_type => "snappy"  #压缩类型
    }
}

4. 检查配置文件是否有语法错

/data/logstash/bin/logstash -f logstash.conf --configtest --verbose

5. 启动logstash

/usr/local/logstash/bin/logstash -f logstash.conf

6. 测试

　　[root@haproxy1 etc]# cat /etc/security/limits.conf >>/var/log/messages

7. 登陆kafka 消息队里服务器

[root@kafka1 ~]# /data/kafka/bin/kafka-topics.sh --list --zookeeper zookeeper1:2181
system-messages #显示topics 为： system-messages

8. 查看system-messages 主题的详情。

[root@kafka1 ~]# /data/kafka/bin/kafka-topics.sh --describe --zookeeper zookeeper1:2181 --topic system-messages
Topic:system-messages PartitionCount:16 ReplicationFactor:1 Configs:
Topic: system-messages Partition: 0 Leader: 3 Replicas: 3 Isr: 3
Topic: system-messages Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: system-messages Partition: 2 Leader: 2 Replicas: 2 Isr: 2
Topic: system-messages Partition: 3 Leader: 3 Replicas: 3 Isr: 3
Topic: system-messages Partition: 4 Leader: 1 Replicas: 1 Isr: 1
Topic: system-messages Partition: 5 Leader: 2 Replicas: 2 Isr: 2
Topic: system-messages Partition: 6 Leader: 3 Replicas: 3 Isr: 3
Topic: system-messages Partition: 7 Leader: 1 Replicas: 1 Isr: 1
Topic: system-messages Partition: 8 Leader: 2 Replicas: 2 Isr: 2
Topic: system-messages Partition: 9 Leader: 3 Replicas: 3 Isr: 3
Topic: system-messages Partition: 10 Leader: 1 Replicas: 1 Isr: 1
Topic: system-messages Partition: 11 Leader: 2 Replicas: 2 Isr: 2
Topic: system-messages Partition: 12 Leader: 3 Replicas: 3 Isr: 3
Topic: system-messages Partition: 13 Leader: 1 Replicas: 1 Isr: 1
Topic: system-messages Partition: 14 Leader: 2 Replicas: 2 Isr: 2
Topic: system-messages Partition: 15 Leader: 3 Replicas: 3 Isr: 3

可以看出，这个主题生成了16个分区，每个分区都有对应自己的Leader，但是我想要有10个分区，3个副本如何办？还是跟我们上面一样命令行来创建主题就行，当然对于logstash输出的我们也可以提前先定义主题，然后启动logstash 直接往定义好的主题写数据就行啦，命令如下：

[root@kafka1 ~]# /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.2.22:2181 --replication-factor 3 --partitions 10 --topic system-messages

logstash 服务器端安装(单独服务器上)

1. 下载

wget https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz

tar -zxvf logstash-2.4.0.tar.gz

mv logstash-2.4.0 /data/

ln -s logstash-2.4.0/ logstash

cd /data/logstash

mkdir logs etc

[root@kafka1etc]# more logstash.conf
input {
    kafka {
        zk_connect => "zookeeper1:2181,zookeeper2:2181,zookeeper3:2181"  #消费者们
        topic_id => "system-messages"
        codec => plain
        reset_beginning => false
        consumer_threads => 5
        decorate_events => true
    }
}
output {
    elasticsearch {
      hosts => ["es1:9200","es2:9200","es3:9200"]
      index => "test-system-messages-%{+YYYY.MM.DD}"          #为了区分之前实验，我这里新生成的所以名字为“test-system-messages-%{+YYYY-MM}”
  }
  }

2 . 启动 logstashserver 服务器进程

[root@logstashserver etc]# /etc/init.d/logstash start

##########################################################

使用redis 替换kafka

logstash 客户端配置文件

input { #这里的输入还是定义的是从日志文件输入
file {
type => "system-messages"
path => "/var/log/messages"
start_position => "beginning"
}
file {
type => "wx-cinyi-com"
path => "/root/wxcinyi.access.log"
start_position => "beginning"
}
}

output {

if [type] == "system-messages" {
redis {
host => "192.168.20.166"
port => "6379"
db => "1"
data_type => "list"
key => "system-messages"
}
}
if [type] == "wx-cinyi-com" {
redis {
host => "192.168.20.166"
port => "6379"
db => "2"
data_type => "list"
key => "wx-cinyi-com"
}
}
}

logstash 服务器端配置文件

input {
if [type] == "system-messages" {
redis {
host => "192.168.20.166"
port => "6379"
db => "1"
data_type => "list"
key => "system-messages"
}
}
if [type] == "wx-cinyi-com" {
redis {
host => "192.168.20.166"
port => "6379"
db => "2"
data_type => "list"
key => "wx-cinyi-com"
}
}
}

output {
if [type] == "system-messages" {
elasticsearch {
hosts => ["es1:9200","es2:9200","es3:9200"]
index => "test-system-messages-%{+YYYY-MM-DD}"
}
}
if [type] == "wx-cinyi-com" {
elasticsearch {
hosts => ["es1:9200","es2:9200","es3:9200"]
index => "wx-cinyi-com-%{+YYYY-MM-DD}"
}
}
}

###########################################################

kibana安装配置

wget https://download.elastic.co/kibana/kibana/kibana-4.6.1-linux-x86_64.tar.gz

[root@kibanai data]# tar -zxvf kibana-4.6.1-linux-x86_64.tar.gz

[root@kibanai data]# mv kibana-4.6.1-linux-x86_64 /data

[root@kibanai data]# ln -s kibana-4.6.1-linux-x86_64 kibana

[root@kibanai data]# /data/kibana/run/

[root@kibanai data]# cd /data/kibana/config/

[root@kibanai config]# cat kibana.yml | grep -v "#" | sed '/^$/d'
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://es1:9200"
elasticsearch.username: "user"
elasticsearch.password: "pass"
elasticsearch.startupTimeout: 5000
pid.file: /data/kibana/run/kibana.pid

安装kibana marvel插件

bin/kibana plugin --install elasticsearch/marvel/latest

3. kibana 启动脚本

#!/bin/bash

KIBBIN='/data/kibana/bin/kibana'
LOCK='/data/kibana/locks'

START() {
if [ -f $LOCK ];then
echo -e "kibana is already \033[32mrunning\033[0m, do nothing."
else
echo -e "Start kibana service.\033[32mdone\033[m"
cd /data/kibana/bin
nohup ./kibana & >/dev/null
touch $LOCK
fi
}

STOP() {
if [ ! -f $LOCK ];then
echo -e "kibana is already stop, do nothing."
else
echo -e "Stop kibana serivce \033[32mdone\033[m"
rm -rf $LOCK
ps -ef | grep kibana | grep -v "grep" | awk '{print $2}' | xargs kill -s 9 >/dev/null
fi
}

STATUS() {
Port=$(netstat -tunl | grep ":5602")
if [ "$Port" != "" ] && [ -f $LOCK ];then
echo -e "kibana is: \033[32mrunning\033[0m..."
else
echo -e "kibana is: \033[31mstopped\033[0m..."
fi
}

case "$1" in
start)
START
;;
stop)
STOP
;;
status)
STATUS
;;
restart)
STOP
sleep 2
START
;;
*)
echo "Usage: /etc/init.d/kibana (|start|stop|status|restart)"
;;
esac