es 如何仅仅存储string es存储方式

转载

我心依旧 2024-07-10 18:02:54

文章标签 es 如何仅仅存储string elasticsearch linux 运维 centos 文章分类 架构后端开发

1. 环境准备

1.1. 硬件&操作系统配置

标准化项	说明	备注
IO设置	写策略设置Write Back，禁用Write Through
BIOS标准设置	开启最大性能模式，选择DAPC或者选择Performance模式	不同的主板配置上有差异
	Memory Frequency（内存频率）选择Maximum Performance模式	不同的主板配置上有差异
OS版本	CentOS Linux 7.x x64	CentOS Linux 7.x以上版本
文件系统	数据存储文件系统类型建议XFS格式
网络配置	使用mode=4模式的网卡绑定	双网卡连接不同的交换机，交换机上需要做适配
数据挂载目录	/data	ES数据库实例的所有文件全部存放在/data目录下；
ntp服务配置	配置ntp服务	要求运维架构部设置
局域网yum源	配置base源、epel源	要求运维架构部设置

1.2. 操作系统软件包&第三方工具

软件包	说明	备注
Head插件
Nodejs	安装head需要，版本node-v6.10.3	4.0以上版本
npm		新版本nodejs包含npm
jdK		1.8以上

1.3. 关闭SELINUX

临时禁用selinux：

# setenforce 0
# more /etc/selinux/config | grep -Ev "$|[#;]"

永久禁用selinux：

# vi /etc/selinux/config

将
SELINUX= enforcing
修改为
SELINUX=disabled 
:x保存退出

1.4. 配置防火墙

如需要禁用防火墙，可执行如下命令：

1）若是CentOS 7.x系统，可执行如下命令禁用防火墙：
关闭防火墙：
# systemctl stop firewalld.service
禁用防火墙开机自启：
# systemctl disable firewalld.service
查看防火墙状态：
# systemctl status firewalld.service

2）若是Ubuntu系统，可执行如下命令禁用防火墙：
# iptables -P INPUT ACCEPT

3）其他系统禁用防火墙命令，请自行查找。

如需配置防火墙规则，可参照如下示例：

# vim /etc/iptables.rules

*filter
:INPUT ACCEPT [3715:523951]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1925:209933]
-A INPUT -i lo -j ACCEPT
#允许ssh端口通过
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
#允许icmp
-A INPUT -p icmp -j ACCEPT
#允许ES服务器间互通，即ES服务器间可互相访问Redis端口为9901~9940的ES实例
-A INPUT -s 192.168.111.111/24 -p tcp -m state --state NEW -m tcp --dport 9901:9940 -j ACCEPT
-A INPUT -s 192.168.111.112/24 -p tcp -m state --state NEW -m tcp --dport 9901:9940 -j ACCEPT
-A INPUT -s 192.168.111.113/24 -p tcp -m state --state NEW -m tcp --dport 9901:9940 -j ACCEPT
#运行其他服务器访问ES，如允许172.172.122.222服务器访问本地ES端口为9901~9940的ES实例
-A INPUT -s 172.172.122.222/32 -p tcp -m state --state NEW -m tcp --dport 9901:9940 -j ACCEPT
#暴露本地22201服务端口
-A INPUT -p tcp -m tcp --dport 22201 -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
COMMIT

防火墙规则配置完毕后，将配置文件导入：

# iptables-restore < /etc/iptables.rules

导入防火墙规则配置文件后，打开防火墙：

# iptables -P INPUT DROP

如需要开机自动恢复规则，把恢复命令添加到启动脚本：

# echo '/sbin/iptables-restore < /etc/iptables.rules' >>/etc/rc.local
# echo '/sbin/iptables -P INPUT DROP' >>/etc/rc.local

# cat /etc/rc.local

1.5. 优化系统参数

编辑系统配置文件sysctl.conf：

# vim /etc/sysctl.conf

vm.overcommit_memory=1
vm.swappiness = 0
vm.max_map_count=655360
net.ipv4.tcp_tw_reuse = 1
fs.file-max = 65536
net.ipv4.ip_local_reserved_ports = 7300,9103,9300,9200

^: net.ipv4.tcp_tw_reuse设置为1表示开启重用，允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭。

^: max_map_count文件包含限制一个进程可以拥有的VMA(虚拟内存区域)的数量。Elasticsearch默认使用mmapfs目录存储其索引。mmap计数的操作系统默认限制可能太低，这可能导致内存不足异常。设置过小，ES会报错“max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]”`

装载ip_conntrack模块：

# modprobe ip_conntrack

载入sysctl配置文件：

# sysctl -p

2. 部署ElasticSearch

2.1. 规划安装目录

规划目录：

# mkdir -p /data/TestES/es/9200/{data,log} 
# mkdir -p /data/es_bak/9200

路径	说明
/data/TestES/es/9200/bin	运行elasticsearch实例和插件管理所需的脚本
/data/TestES/es/9200/data	elasticsearch使用的所有数据的存储位置
/data/TestES/es/9200/lib	存放es 的lib jar包
/data/TestES/es/9200/logs	关于事件和错误记录的文件，es日志、慢查询等
/data/TestES/es/9200/modules	Es的模块
/data/TestES/es/9200/plugins	存放es第三方插件，如分词插件，s3插件等
/data/TestES/es/9200/config	配置文件所在的目录

安装介质目录（全部根据实例独立部署）：
安装介质目录只存放安装介质，目录名：$port/bin，不存放任何配置文件、数据文件等。

# tree -L 1 $port/bin | more
 ├── elasticsearch
 ├── elasticsearch.bat
 ├── elasticsearch-env
 ├── elasticsearch-env.bat
 ├── elasticsearch-keystore
 ├── elasticsearch-keystore.bat
 ├── elasticsearch-plugin
 ├── elasticsearch-plugin.bat
 ├── elasticsearch-service.bat
 ├── elasticsearch-service-mgr.exe
 ├── elasticsearch-service-x64.exe
 ├── elasticsearch-translog
 └── elasticsearch-translog.bat

2.2. 创建用户及用户组

# groupadd es
# useradd -d /home/es -g es  es
# passwd es
输入密码:rQ0nmY9IQuTcYXc5
输入密码:rQ0nmY9IQuTcYXc5

^: -s /sbin/nologin，不允许用户登陆。

2.3. 调整用户组资源限制

在 Linux(Unix)系统中，因为 ulimit 命令的存在，会对程序使用操作系统资源进行限制。为了使mysql数据库能够正常运行，建议用户检查并调整的es安装用户的系统资源。

# grep ^es /etc/security/limits.conf

es soft fsize unlimited
es hard fsize unlimited
es soft cpu unlimited
es hard cpu unlimited
es soft as unlimited
es hard as unlimited
es soft nofile 655350
es hard nofile 655350
es soft nproc 65535
es hard nproc 65535
es soft memlock unlimited
es hard memlock unlimited

资源名	限制	备注
fsize	unlimited	限定进程所写文件的大小
cpu	unlimited	限定进程能占用的CPU时间
as	unlimited	限制进程的地址空间（adress space）大小
nofile	65536	限制进程所能打开的文件数量。Elasticsearch需要使用许多文件描述符或文件句柄。文件描述符用完可能是灾难性的，很可能会导致数据丢失。确保将运行Elasticsearch的用户的打开文件描述符的数量限制增加到65536或更高。
nproc	65536	限制某个用户所能打开的最多进程&线程数量。Elasticsearch为不同类型的操作使用许多线程池，重要的是，它能够在需要时创建新的线程。确保Elasticsearch用户可以创建的线程数至少为4096。
memlock	unlimited	最大锁定内存地址空间。在Linux/Unix系统上使用mlockall，尝试将进程地址空间锁定到RAM中，防止任何Elasticsearch内存被交换出去

2.4. 安装JDK

从Elasticsearch 5开始需要JAVA 8以上的版本，Elasticsearch 从6.5开始支持JAVA 11，Elasticsearch 7.0开始内置了Java环境。

2.4.1. 方法一：RPM包安装

# rpm -ivh jdk-8u172-linux-x64.rpm
# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

2.4.2. 方法二：压缩包解压

解压安装包

# cd /opt/app/softwares
# tar zxvf java-1.8.0-openjdk.linux.x86_64.tar.gz -C /var/lib/
# cd /var/lib
# mv java-1.8.0-openjdk.linux.x86_64 jdk1.8

配置环境变量

# vim /etc/profile

export JAVA_HOME=/var/lib/jdk1.8
export JRE_HOME=$JAVA_HOME/jre
export JAVA_PATH=$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_PATH:

# source /etc/profile
# java -version

^: JDK下载地址： http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

2.5. 安装ES

与Elasticsearch交互的主要接口是基于HTTP协议和REST的。这意味着你甚至可以使用Web浏览器来完成基本的查询和请求，但对于更复杂的情况，你需要额外的命令行工具，比如curl。如果你使用Linux或OS X命令，curl已经可用了。如果你使用Windows，可以从http://curl.haxx.se/download.html下载。

上传elasticsearch-7.6.0-linux-x86_64.tar.gz至/tmp目录

# cd /opt/app/softwares/
# tar -xvzf elasticsearch-6.1.1.tar.gz
# cp -af /opt/app/softwares/elasticsearch-6.1.1/* /data/TestES/es/9200/

^: Elasticsearch 二进制文件下载地址： https://www.elastic.co/downloads/elasticsearch

2.6. 修改配置文件elasticsearch.yml

2.6.1. 节点192.168.111.111配置信息

# vim /data/TestES/es/9200/config/elasticsearch.yml

cluster.name: test_es
node.name: 192.168.111.111-9200
node.master: true
node.data: true
path.data: /data/TestES/es/9200/data
path.logs: /data/TestES/es/9200/log
path.repo: ["/data/es_bak/9200"]

bootstrap.memory_lock: true
bootstrap.system_call_filter: false
network.host: 192.168.111.111
http.port: 9200
transport.tcp.port: 7300
discovery.zen.ping.unicast.hosts: ["192.168.111.111:7300","192.168.111.112:7300","192.168.111.113:7300"]
http.cors.enabled: true
http.cors.allow-origin: "*"
script.max_compilations_rate: 10000000/1m

discovery.zen.minimum_master_nodes: 2
cluster.routing.allocation.same_shard.host: true

thread_pool.search.size: 96
thread_pool.search.queue_size: 2000
thread_pool.search.max_queue_size: 2000
thread_pool.search.min_queue_size: 2000
thread_pool.search.target_response_time: 1s
thread_pool.search.auto_queue_frame_size: 2000

2.6.2. 节点192.168.111.112配置信息

# vim /data/TestES/es/9200/config/elasticsearch.yml

cluster.name: test_es
node.name: 192.168.111.112-9200
node.master: true
node.data: true
path.data: /data/TestES/es/9200/data
path.logs: /data/TestES/es/9200/log
path.repo: ["/data/es_bak/9200"]

bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 192.168.111.112
http.port: 9200
transport.tcp.port: 7300
discovery.zen.ping.unicast.hosts: ["192.168.111.111:7300","192.168.111.112:7300","192.168.111.113:7300"]
http.cors.enabled: true
http.cors.allow-origin: "*"
script.max_compilations_rate: 10000000/1m

discovery.zen.minimum_master_nodes: 2
cluster.routing.allocation.same_shard.host: true

thread_pool.search.size: 96
thread_pool.search.queue_size: 2000
thread_pool.search.max_queue_size: 2000
thread_pool.search.min_queue_size: 2000
thread_pool.search.target_response_time: 1s
thread_pool.search.auto_queue_frame_size: 2000

2.6.3. 节点192.168.111.113配置信息

# vim /data/TestES/es/9200/config/elasticsearch.yml

cluster.name: test_es
node.name: 192.168.111.113-9200
node.master: true
node.data: true
path.data: /data/TestES/es/9200/data
path.logs: /data/TestES/es/9200/log
path.repo: ["/data/es_bak/9200"]

bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 192.168.111.113
http.port: 9200
transport.tcp.port: 7300
discovery.zen.ping.unicast.hosts: ["192.168.111.111:7300","192.168.111.112:7300","192.168.111.113:7300"]
http.cors.enabled: true
http.cors.allow-origin: "*"
script.max_compilations_rate: 10000000/1m

discovery.zen.minimum_master_nodes: 2
cluster.routing.allocation.same_shard.host: true

thread_pool.search.size: 96
thread_pool.search.queue_size: 2000
thread_pool.search.max_queue_size: 2000
thread_pool.search.min_queue_size: 2000
thread_pool.search.target_response_time: 1s
thread_pool.search.auto_queue_frame_size: 2000

2.7. 修改JVM配置

elasticsearch 7.1下载的默认设置是1GB。建议：Xmx和Xms设置成一样，Xmx不要超过机器内存的50%，并且不要超过30GB。

2.7.1. 节点192.168.111.111配置信息

根据实际需求配置

# vim /data/TestES/es/9200/config/jvm.options
 -Xms30g
 -Xmx30g

# grep -E '^-Xms|^-Xmx' /data/TestES/es/9200/config/jvm.options
-Xms30g
-Xmx30g

2.7.2. 节点192.168.111.112配置信息

根据实际需求配置

# vim /data/TestES/es/9200/config/jvm.options
 -Xms30g
 -Xmx30g

# grep -E '^-Xms|^-Xmx' /data/TestES/es/9200/config/jvm.options
-Xms30g
-Xmx30g

2.7.3. 节点192.168.111.113配置信息

根据实际需求配置

# vim /data/TestES/es/9200/config/jvm.options
 -Xms30g
 -Xmx30g

# grep -E '^-Xms|^-Xmx' /data/TestES/es/9200/config/jvm.options
-Xms30g
-Xmx30g

2.8. 安装ElasticSearch分词器

Elasticsearch提供插件的机制对系统进行了扩展，如Discovery Plugiin、Analysis Plugin、Security Plugin、Management Plugin、Ingest Plugin、Mapper Plugin、Backup Plugin。

在线安装：
# cd /data/TestES/es/9200/
# bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.1.1/elasticsearch-analysis-ik-6.1.1.zip
-> Downloading https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.1.1/elasticsearch-analysis-ik-6.1.1.zip
[=================================================] 100%   
-> Installed analysis-ik

查看插件
# /data/TestES/es/9200/bin/elasticsearch-plugin list
analysis-ik

若不能在线安装，则需要手动下载后再行安装：
下载地址：https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.1.1/elasticsearch-analysis-ik-6.1.1.zip
下载后上传到/opt/app/softwares/目录：

# cd /opt/app/softwares/
# unzip elasticsearch-analysis-ik-6.1.1.zip
# mv elasticsearch /data/TestES/es/9200/plugins/analysis-ik

2.9. 修改权限

# chown -R es:es /data/TestES/es
# chown -R es:es /data/es_bak/9200

2.10. 启动es实例

启动实例：

# su - es -c 'cd /data/TestES/es/9200/bin && ./elasticsearch -d'
 
 # ps -ef|grep -v grep|grep "elasticsearch.bootstrap.Elasticsearch -d"

^: 对于报错could not find java; set JAVA_HOME or ensure java is in PATH，建议采用yum安装JAVA，因为它优先在/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin等目录寻找java程序。

2.11. 检查运行情况

检查es6.1.1集群的节点信息：
# curl http://192.168.111.111:9200
{
  "name" : "192.168.111.111-9200",
  "cluster_name" : "test_es",
  "cluster_uuid" : "z7lzGOndSOavBD7mp-C2LA",
  "version" : {
    "number" : "6.1.1",
    "build_hash" : "bd92e7f",
    "build_date" : "2017-12-17T20:23:25.338Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}


# curl http://192.168.111.112:9200
{
  "name" : "192.168.111.112-9200",
  "cluster_name" : "test_es",
  "cluster_uuid" : "z7lzGOndSOavBD7mp-C2LA",
  "version" : {
    "number" : "6.1.1",
    "build_hash" : "bd92e7f",
    "build_date" : "2017-12-17T20:23:25.338Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}


# curl http://192.168.111.113:9200
{
  "name" : "192.168.111.113-9200",
  "cluster_name" : "test_es",
  "cluster_uuid" : "z7lzGOndSOavBD7mp-C2LA",
  "version" : {
    "number" : "6.1.1",
    "build_hash" : "bd92e7f",
    "build_date" : "2017-12-17T20:23:25.338Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}


# curl http://192.168.111.111:9200/_cat/health?v
epoch      timestamp cluster              status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1620799827 14:10:27  test_es green           3         3      0   0    0    0        0             0                  -                100.0%


# curl http://192.168.111.112:9200/_cat/health?v
epoch      timestamp cluster              status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1620799869 14:11:09  test_es green           3         3      0   0    0    0        0             0                  -                100.0%



# curl http://192.168.111.113:9200/_cat/health?v
epoch      timestamp cluster              status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1620799982 14:13:02  test_es green           3         3      0   0    0    0        0             0                  -                100.0%


# curl http://192.168.111.111:9200/_cat/nodes?v
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.111            5          35   0    0.02    0.06     0.05 mdi       *      192.168.111.111-9200
192.168.111.113            3          97   2    1.04    0.70     0.69 mdi       -      192.168.111.113-9200
192.168.111.112            3          74   0    0.00    0.01     0.05 mdi       -      192.168.111.112-9200


# curl http://192.168.111.112:9200/_cat/nodes?v
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.113            3          97   2    0.73    0.76     0.71 mdi       -      192.168.111.113-9200
192.168.111.111            5          35   0    0.04    0.05     0.05 mdi       *      192.168.111.111-9200
192.168.111.112            3          74   0    0.01    0.02     0.05 mdi       -      192.168.111.112-9200


# curl http://192.168.111.113:9200/_cat/nodes?v
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.113            3          97   2    0.60    0.73     0.70 mdi       -      192.168.111.113-9200
192.168.111.111            5          35   0    0.03    0.04     0.05 mdi       *      192.168.111.111-9200
192.168.111.112            3          74   0    0.00    0.01     0.05 mdi       -      192.168.111.112-9200



# curl -XGET 'http://192.168.111.111:9200/_nodes/stats?pretty'
# curl -XGET 'http://192.168.111.112:9200/_nodes/stats?pretty'
# curl -XGET 'http://192.168.111.113:9200/_nodes/stats?pretty'


查看es6.1.1集群的状态：
# curl http://192.168.111.111:9200/_cluster/health?pretty -o cg.txt
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   473  100   473    0     0   9385      0 --:--:-- --:--:-- --:--:--  9460


# cat cg.txt
{
  "cluster_name" : "test_es", //集群名称
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,				//节点数
  "number_of_data_nodes" : 3,			//数据节点数
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}



查看日志：
# tail -1000f /data/TestES/es/9200/log/test_es.log
[2021-05-12T14:03:02,611][INFO ][o.e.n.Node               ] [192.168.111.111-9200] initializing ...
[2021-05-12T14:03:02,719][INFO ][o.e.e.NodeEnvironment    ] [192.168.111.111-9200] using [1] data paths, mounts [[/data (/dev/sdb1)]], net usable_space [1.5tb], net total_space [1.7tb], types [ext4]
[2021-05-12T14:03:02,720][INFO ][o.e.e.NodeEnvironment    ] [192.168.111.111-9200] heap size [29.7gb], compressed ordinary object pointers [true]
[2021-05-12T14:03:02,722][INFO ][o.e.n.Node               ] [192.168.111.111-9200] node name [192.168.111.111-9200], node ID [5rU8v84KRvGPKhu1pDz4_A]
[2021-05-12T14:03:02,722][INFO ][o.e.n.Node               ] [192.168.111.111-9200] version[6.1.1], pid[1245], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Linux/3.10.0-862.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2021-05-12T14:03:02,722][INFO ][o.e.n.Node               ] [192.168.111.111-9200] JVM arguments [-Xms30g, -Xmx30g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/data/TestES/es/9200, -Des.path.conf=/data/TestES/es/9200/config]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [aggs-matrix-stats]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [analysis-common]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [ingest-common]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [lang-expression]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [lang-mustache]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [lang-painless]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [mapper-extras]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [parent-join]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [percolator]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [reindex]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [repository-url]
[2021-05-12T14:03:04,027][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [transport-netty4]
[2021-05-12T14:03:04,028][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded module [tribe]
[2021-05-12T14:03:04,028][INFO ][o.e.p.PluginsService     ] [192.168.111.111-9200] loaded plugin [analysis-ik]
[2021-05-12T14:03:05,564][INFO ][o.e.d.DiscoveryModule    ] [192.168.111.111-9200] using discovery type [zen]
[2021-05-12T14:03:06,165][INFO ][o.e.n.Node               ] [192.168.111.111-9200] initialized
[2021-05-12T14:03:06,166][INFO ][o.e.n.Node               ] [192.168.111.111-9200] starting ...
[2021-05-12T14:03:06,452][INFO ][o.e.t.TransportService   ] [192.168.111.111-9200] publish_address {192.168.111.111:7300}, bound_addresses {192.168.111.111:7300}
[2021-05-12T14:03:06,462][INFO ][o.e.b.BootstrapChecks    ] [192.168.111.111-9200] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2021-05-12T14:03:08,169][INFO ][o.e.m.j.JvmGcMonitorService] [192.168.111.111-9200] [gc][2] overhead, spent [280ms] collecting in the last [1s]
[2021-05-12T14:03:09,539][INFO ][o.e.c.s.MasterService    ] [192.168.111.111-9200] zen-disco-elected-as-master ([1] nodes joined)[{192.168.111.112-9200}{x7ewphCFQsm_h5O7v7FGDA}{dsktnkL_Rp6UwIsblBuzOw}{192.168.111.112}{192.168.111.112:7300}], reason: new_master {192.168.111.111-9200}{5rU8v84KRvGPKhu1pDz4_A}{Ssu0lafyTCewi-TY8MQ1Uw}{192.168.111.111}{192.168.111.111:7300}, added {{192.168.111.112-9200}{x7ewphCFQsm_h5O7v7FGDA}{dsktnkL_Rp6UwIsblBuzOw}{192.168.111.112}{192.168.111.112:7300},}
[2021-05-12T14:03:09,602][INFO ][o.e.c.s.ClusterApplierService] [192.168.111.111-9200] new_master {192.168.111.111-9200}{5rU8v84KRvGPKhu1pDz4_A}{Ssu0lafyTCewi-TY8MQ1Uw}{192.168.111.111}{192.168.111.111:7300}, added {{192.168.111.112-9200}{x7ewphCFQsm_h5O7v7FGDA}{dsktnkL_Rp6UwIsblBuzOw}{192.168.111.112}{192.168.111.112:7300},}, reason: apply cluster state (from master [master {192.168.111.111-9200}{5rU8v84KRvGPKhu1pDz4_A}{Ssu0lafyTCewi-TY8MQ1Uw}{192.168.111.111}{192.168.111.111:7300} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[{192.168.111.112-9200}{x7ewphCFQsm_h5O7v7FGDA}{dsktnkL_Rp6UwIsblBuzOw}{192.168.111.112}{192.168.111.112:7300}]]])
[2021-05-12T14:03:09,617][INFO ][o.e.h.n.Netty4HttpServerTransport] [192.168.111.111-9200] publish_address {192.168.111.111:9200}, bound_addresses {192.168.111.111:9200}
[2021-05-12T14:03:09,617][INFO ][o.e.n.Node               ] [192.168.111.111-9200] started
[2021-05-12T14:03:09,708][INFO ][o.e.g.GatewayService     ] [192.168.111.111-9200] recovered [0] indices into cluster_state
[2021-05-12T14:03:14,601][INFO ][o.e.c.s.MasterService    ] [192.168.111.111-9200] zen-disco-node-join[{192.168.111.113-9200}{ePwPxinKSmOZnHpee4YA1Q}{BSaA1RUGSoi0ovlewaYtFg}{192.168.111.113}{192.168.111.113:7300}], reason: added {{192.168.111.113-9200}{ePwPxinKSmOZnHpee4YA1Q}{BSaA1RUGSoi0ovlewaYtFg}{192.168.111.113}{192.168.111.113:7300},}
[2021-05-12T14:03:14,882][INFO ][o.e.c.s.ClusterApplierService] [192.168.111.111-9200] added {{192.168.111.113-9200}{ePwPxinKSmOZnHpee4YA1Q}{BSaA1RUGSoi0ovlewaYtFg}{192.168.111.113}{192.168.111.113:7300},}, reason: apply cluster state (from master [master {192.168.111.111-9200}{5rU8v84KRvGPKhu1pDz4_A}{Ssu0lafyTCewi-TY8MQ1Uw}{192.168.111.111}{192.168.111.111:7300} committed version [3] source [zen-disco-node-join[{192.168.111.113-9200}{ePwPxinKSmOZnHpee4YA1Q}{BSaA1RUGSoi0ovlewaYtFg}{192.168.111.113}{192.168.111.113:7300}]]])

3. 安装Head

3.1. 创建目录

# mkdir -p /data/TestES/{es-head,nodejs}

3.2. 安装elasticsearch-head插件

从官网https://nodejs.org/en/download/下载elasticsearch-head-master.zip包：

安装方法一：
# wget https://github.com/mobz/elasticsearch-head/archive/master.zip
# unzip master.zip
# mv elasticsearch-head-master /data/TestES/es-head/

# chown -R es:es /data/TestES/es-head/
# chown -R es:es /data/TestES/nodejs/

安装方法二：
手动下载elasticsearch-head-master.zip上传到/tmp目录并解压：
下载地址：https://github.com/mobz/elasticsearch-head/archive/elasticsearch-head-master.zip
# unzip elasticsearch-head-master.zip -d /data/TestES/es-head/
# ls -l /data/TestES/es-head/elasticsearch-head-master/

# chown -R es:es /data/TestES/es-head/
# chown -R es:es /data/TestES/nodejs/

安装方法三：
# cd /data/TestES/es-head/
# yum install –y git
# git clone https://github.com/mobz/elasticsearch-head.git

3.3. 安装node

由于head插件本质上还是一个nodejs的工程，因此需要安装node，使用npm来安装依赖的包。

# wget https://nodejs.org/dist/v10.15.0/node-v10.15.0-linux-x64.tar.gz

# tar -xzvf node-v10.15.0-linux-x64.tar.gz -C /data/TestES/nodejs
# ls -l /data/TestES/nodejs/node-v10.15.0-linux-x64
total 164
drwxrwxr-x 2 500 500  4096 Dec 26  2018 bin
-rw-rw-r-- 1 500 500 52896 Dec 26  2018 CHANGELOG.md
drwxrwxr-x 3 500 500  4096 Dec 26  2018 include
drwxrwxr-x 3 500 500  4096 Dec 26  2018 lib
-rw-rw-r-- 1 500 500 65839 Dec 26  2018 LICENSE
-rw-rw-r-- 1 500 500 25981 Dec 26  2018 README.md
drwxrwxr-x 5 500 500  4096 Dec 26  2018 share

# chown -R es:es /data/TestES/es-head/
# chown -R es:es /data/TestES/nodejs/

3.4. 配置环境变量

# vim /etc/profile

export NODE_HOME=/data/TestES/nodejs/node-v10.15.0-linux-x64
export PATH=$PATH:$NODE_HOME/bin
export NODE_PATH=$NODE_HOME/lib/node_modules

修改完配置变量运行下面命令使其生效：

# source /etc/profile

验证是否生效：

# node -v
v10.15.0
# npm -v
6.4.1

3.5. 安装grunt

grunt是基于Node.js的项目构建工具，可以进行打包压缩、测试、执行等等工作，elasticsearch-head插件就是通过grunt启动的。

在192.168.111.111节点配置：

# cd /data/TestES/es-head/elasticsearch-head-master
# npm install -g grunt-cli
/data/TestES/nodejs/node-v10.15.0-linux-x64/bin/grunt -> /data/TestES/nodejs/node-v10.15.0-linux-x64/lib/node_modules/grunt/bin/grunt
+ grunt@1.4.0
added 102 packages from 109 contributors in 83.629s

# grunt -version
grunt-cli v1.4.2
grunt v1.4.0

3.6. 安装node模块

在192.168.111.111节点配置：
# cd /data/TestES/es-head/elasticsearch-head-master
# npm install

npm WARN deprecated coffee-script@1.10.0: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated http2@3.3.7: Use the built-in module in node 9.0.0 or newer, instead
npm WARN deprecated chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated core-js@2.6.12: core-js@<3.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
npm WARN deprecated phantomjs-prebuilt@2.1.16: this package is now deprecated
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
npm WARN deprecated json3@3.2.6: Please use the native JSON object instead of JSON 3
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated

> phantomjs-prebuilt@2.1.16 install /data/TestES/es-head/elasticsearch-head-master/node_modules/phantomjs-prebuilt
> node install.js

PhantomJS not found on PATH
Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
Receiving...
  [========================================] 100%

Received 22866K total.
Extracting tar contents (via spawned process)
Removing /data/TestES/es-head/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom
Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1620802417724/phantomjs-2.1.1-linux-x86_64 -> /data/TestES/es-head/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom
chmod failed: phantomjs was not successfully copied to /data/TestES/es-head/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! phantomjs-prebuilt@2.1.16 install: `node install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the phantomjs-prebuilt@2.1.16 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-05-12T06_53_41_147Z-debug.log


# npm install -g cnpm --registry=https://registry.npm.taobao.org
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
/data/TestES/nodejs/node-v10.15.0-linux-x64/bin/cnpm -> /data/TestES/nodejs/node-v10.15.0-linux-x64/lib/node_modules/cnpm/bin/cnpm
+ cnpm@6.2.0
added 700 packages from 975 contributors in 19.397s

3.7. 修改Gruntfile.js

# vi /data/TestES/es-head/elasticsearch-head-master/Gruntfile.js

添加hostname: '*'，如下图：
                options: {
                        port: 9300,				//需要修改端口号
                        base: '.',
                        keepalive: true,
                        hostname: '*'
                }
        }
}

# grep -E 'hostname|keepalive|port' /data/TestES/es-head/elasticsearch-head-master/Gruntfile.js
module.exports = function(grunt) {
					port: 9300,
					keepalive: true,
					hostname: '*'

3.8. 修改app.js

# cd /data/TestES/es-head/elasticsearch-head-master
# vi _site/app.js 
init: function(parent) {
        this._super();
        this.prefs = services.Preferences.instance();
        this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.111.111:9200";   
        if( this.base_uri.charAt( this.base_uri.length - 1 ) !== "/" ) {
                // XHR request fails if the URL is not ending with a "/"
                this.base_uri += "/";
        }
 
 
 # grep "this.base_uri = this.config.base_uri " /data/TestES/es-head/elasticsearch-head-master/_site/app.js
 			this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.111.111:9200";

3.9. 启动head

# cd /data/TestES/es-head/elasticsearch-head-master/node_modules/grunt/bin 
# nohup ./grunt server  > /tmp/grunt_server.log 2>&1 &

# nohup grunt server > /tmp/grunt_server.log 2>&1 &
# tail -100f /tmp/grunt_server.log
nohup: ignoring input
>> Local Npm module "grunt-contrib-jasmine" not found. Is it installed?

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9300

或者：编辑启动脚本：
# vi starthead.sh
#!/bin/bash
cd /data/TestES/es-head/elasticsearch-head-master
nohup grunt server > /tmp/grunt_server.log 2>&1 &

# chmod +x starthead.sh
# chown -R es:es /data/TestES/es-head/elasticsearch-head-master

# su - es
# cd /data/TestES/es-head/elasticsearch-head-master
# ./starthead.sh
# ls -l /tmp/grunt_server.log
-rw-rw-r-- 1 es es 184 May 26 16:20 /tmp/grunt_server.log

4. 附录

4.1. 文件目录规划

集群配置参数文件：/data/TestES/es/9200/config/elasticsearch.yml
内存参数文件：/data/TestES/es/9200/config/jvm.options

4.2. 配置参数规划

基本参数为除了路径参数以外，在数据库使用前必须要设置的参数。

4.2.1. 基本参数

参数名称	说明	缺省值	推荐值
cluster.name	集群名称
node.name	节点名称
node.master	候选主节点
node.data	数据节点
node.ingest	数据过滤转换节点
network.host		127.0.0.1	集群需要指定ip
network.bind_host	设置network.host就无需设置此参数
network.publish_host	设置network.host就无需设置此参数
http.port	http端口	9200
transport.tcp.port	Tcp业务访问端口	9300
discovery.zen.ping.unicast.hosts	单播地址
discovery.zen.minimum_master_nodes	最少master节点数	1	根据高可用来设置
cluster.routing.allocation.same_shard.host	禁止相同分片在同一主机	false	true
discovery.zen.ping.multicast.enabled	禁用多播	true	false
transport.tcp.compress	压缩tcp传输时的数据	false
http.max_content_length	设置请求内容的最大容量	默认100mb
index.translog.flush_threshold_ops	当事务日志累积到多少条数据后flush一次

4.2.2. 路径参数

参数名称	说明	缺省值	推荐值
path.data	Data路径		{port}/data/
path.logs	日志路径		{port}/logs/
path.repo	快照路径		/backup/

4.2.3. 内存参数

参数名称	说明	缺省值	推荐值
-Xms	最小内存	1g	31g
-Xmx	最大内存	1g	31g
-Des.allow_insecure_settings	扩展对象存储参数	False	true

4.2.4. 其它参数

参数名称	说明	缺省值	推荐值
thread_pool.index.queue_size
thread_pool.bulk.queue_size
bootstrap.memory_lock	锁定物理内存地址，防止es内存被交换出去	false	true
bootstrap.system_call_filter	centos6.x操作系统不支持SecComp，所以导致检测失败，失败后直接导致ES不能启动。	true	false
bootstrap.mlockall	锁定内存	False	true
script.max_compilations_rate	每分钟将编译内联脚本个数	默认情况下，每分钟将编译最多15个内联脚本	10000000/1m
http.cors.allow-origin	当设置允许跨域，默认为*,表示支持所有域名，如果我们只是允许某些网站能访问，那么可以使用正则表达式。比如只允许本地地址
http.cors.enabled	是否支持跨域	False	true
indices.fielddata.cache.size	达到该阈值，就会把旧数据交换出去。该参数可以设置百分比或者绝对值。	默认设置是不限制	设定内存的10%
indices.breaker.fielddata.limit	fielddata 到 limit 阈值告警	JVM堆内存的60%	JVM堆内存的60%
discovery.zen.ping.timeout	延长默认超时时间	3s	30s
discovery.zen.fd.ping_timeout
discovery.zen.fd.ping.interval
discovery.zen.fd.ping.retries
thread_pool.search.size	控制线程数	默认为生产cpu核数的2-3倍	96
thread_pool.search. queue_size	请求队列的初始大小		2000
thread_pool.search.max_queue_size _size	queue_size可以调整的最大值		2000
thread_pool.search.min_queue_size _size	queue_size可以调整的最小值		2000
thread_pool.search.target_response_time	线程池任务目标平均响应时间，如果任务通常超过这时间，则拒绝任务	这个根据业务需要调整	1s
thread_pool.search. auto_queue_frame_size			2000

4.3. 配置文件样例

ES配置文件规范

# vim /data/TestES/es/9200/config/elasticsearch.yml

# 集群名称
cluster.name: test_es
# 节点名称
node.name: 192.168.111.112-9200
# 候选主节点  
node.master: false
# 数据节点
node.data: true 
# Data路径
path.data:  /data/TestES/es/9200/data 
# 日志路径  
path.logs:   /data/TestES/es/9200/logs
# 快照路径  
path.repo: /backup/9200

#开启内存锁
bootstrap.memory_lock: true 
bootstrap.system_call_filter: false 
# ip地址
network.host: 192.168.111.112
# 服务端口号
http.port: 9200 
# 集群间通信端口号
transport.tcp.port: 19200

# 6.X版本中参数，配置集群发现master节点，单播地址 
discovery.zen.ping.unicast.hosts: ["192.168.111.111:20302","192.168.111.112:20302","192.168.111.113:20302"]
# 7.X版本中参数，discovery.seed_hosts对应旧版中的discovery.zen.ping.unicast.hosts，配置集群发现master节点
discovery.seed_hosts: ["192.168.111.111:20302","192.168.111.112:20302","192.168.111.113:20302"]
#初始化主节点列表配置
cluster.initial_master_nodes: ["192.168.111.111-9200","192.168.111.112-9200"]
 
#开启跨域访问，EShead安装需开启
http.cors.enabled: true 
http.cors.allow-origin: "*" 
script.max_compilations_rate: 10000000/1m 

discovery.zen.minimum_master_nodes: 2 
#  延长默认超时时间
discovery.zen.ping.timeout: 30 
cluster.routing.allocation.same_shard.host: true 

thread_pool: 
search: 
# 控制线程数， 默认为生产cpu核数的2-3倍
size: 96 
# 请求队列的初始大小
queue_size: 2000 
# queue_size可以调整的最小值  
min_queue_size: 2000 
max_queue_size: 2000 
auto_queue_frame_size: 2000 
#  线程池任务目标平均响应时间，如果任务通常超过这时间，则拒绝任务  
target_response_time: 1s

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：mysql api借口 mysql的api

下一篇：stm32精英freemodbus

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯