前言:由于项目需求,近期要用到Elasticsearch,故此仅以此随笔记下入门足
1. Elasticsearch
- 下载 :https://www.elastic.co/cn/downloads/elasticsearch (可以点击“past releases”选择过去的版本)
- 安装 : 将压缩包解压,进入elasticsearch的bin目录,双击elasticsearch.bat启动服务,默认端口是9200;
- 确认: 启动完成之后,在浏览器中访问http://localhost:9200/
- 更多:Elasticsearch 安装细节
2. kibana 安装下载 (Kibana是一个为ElasticSearch 提供的数据分析的 Web 接口。可使用它对日志进行高效的搜索、可视化、分析等各种操作。)
- 下载:https://www.elastic.co/cn/downloads/kibana (点击“past releases”选择Elasticsearch的版本)
- 安装:将压缩包解压,进入kibana 的bin目录,双击kibana.bat 启动服务,(运行之前先启动ES)
- 确认:启动完成后,在浏览器访问 http://127.0.0.1:5601/app/kibana#/dev_tools/console?_g=() 进行操作
- 更多: 在Dev Tools 下操作
#查看索引
GET locations/_search
#查看索引数据总数
GET /locations/_count
#统计总存储空间占用
GET /_cat/shards?v
#创建索引
PUT /{IndexName}?pretty
#索引取别名
PUT /{OldIndexName}/_alias/{NewIndexName}
#查看某个索引映射
GET locations/_mapping
# 查询所有索引信息
GET /_cat/indices?v
#删除locations索引
DELETE /locations?pretty
3. springboot整合
- 引入依赖
<!--elasticsearch-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
<version>3.2.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.10</version>
</dependency>
- 添加配置
#elasticsearch配置
spring.data.cassandra.cluster-name=elasticsearch //默认集群名称
spring.data.elasticsearch.cluster-nodes =127.0.0.1:9300 //默认启动集群节点
- 实体类配置
@Data注释只是为了添加get和set方法
indexName 索引名称(理解为mysql数据库名) , type 索引类型名称(理解为表名)
听说: 索引文档实体 注意此时@Id注解的导入包来自import org.springframework.data.annotation.Id,索引名称必须为小写
@Data
@Document(indexName = "locations",type = "tbCidEntity")
public class TbCidEntity {
@Id
private String keyCid;
@Field(type = FieldType.Text)
private String mcc;
@Field(type = FieldType.Keyword)
private String pos;
@Field(type = FieldType.Text,index = false)
private String remark;
@Field(type = FieldType.Date, format = DateFormat.custom,pattern = "yyyy-MM-dd")
@JsonFormat(shape = JsonFormat.Shape.STRING, pattern ="yyyy-MM-dd",timezone="GMT+8")
private Date mdfTm;
@Field(type = FieldType.Text,index = false)
private String mdfBy;
@Field(type = FieldType.Date, format = DateFormat.custom,pattern = "yyyy-MM-dd")
}
elasticsearch资源库 继承了ElasticsearchRepository,封装了很多API(TbCid是索引文档实体,String是文档id类型)
@Component
public interface TBCidRepository extends ElasticsearchRepository <TbCid,String>{
TbCid findByKeyCid(String keycid);
}
- 工具类使用
@Autowired
private TBCidRepository tbCidRepository;
//从es中根据cid查询tbcid
public TbCid findByESCid(String cid){
BoolQueryBuilder builder = QueryBuilders.boolQuery();
MatchPhraseQueryBuilder cid1 = QueryBuilders.matchPhraseQuery("cid", cid);
builder.must(cid1);
Iterable<TbCid> search = tbCidRepository.search(builder);
Iterator<TbCid> iterator = search.iterator();
if (iterator.hasNext()){
TbCid next = iterator.next();
return next;
}else {
return null;
}
}
//从es中根据keycid查询tbcid
public TbCid findByESKeyCid(String cid){
return tbCidRepository.findByKeyCid(cid);
}
//从将对象存入es
public void saveEntityToEs( TbCid cid){
tbCidRepository.save(cid);
}
4. 新增后在 kibana 的 Dev Tools 下 检查
GET /locations/tbCidEntity/_search
查询结果:
{ "_index": "locations",
"_type": "tbCidEntity",
"_id": "1",
"_score": 1,
"_source": { ...} }
4. 总结
- 好像es和springboot整合存在许多兼容问题,我这里是springboot2才兼容
5. linux 出现问题
- 问题 java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
解决办法:在所有节点的elasticsearch.yml配置文件中加入:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
2. 问题:内存不够
解决办法添加虚拟内存
3. 问题:max number of threads [3818] for user [es] is too low, increase to at least [4096]
最大线程个数太低。修改配置文件/etc/security/limits.conf,增加配置,用户退出后重新登录生效
* soft nproc 4096
* hard nproc 4096
4. 问题:max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
修改/etc/security/limits.conf文件,增加配置,用户退出后重新登录生效
* soft nofile 65536
* hard nofile 65536
注:3、4问题 为每个进程最大同时打开文件数太小,可通过下面2个命令查看当前数量
ulimit -Hn
ulimit -Sn
5. 问题:max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
修改/etc/sysctl.conf文件,增加配置vm.max_map_count=262144
vi /etc/sysctl.conf
sysctl -p
执行命令sysctl -p生效
6. 问题:[1]: max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
错误原因:启动检查未通过, elasticsearch用户的最大线程数太低
90-nproc.conf 为
* soft nproc 2048
root soft nproc unlimited
6.后端启动
./elasticsearch -d
稍等片刻,再次访问地址,还是相同结果
此时,如果想关闭服务
[esuser@localhost bin]$ ps -ef|grep esuser
esuser 3318 3007 0 Feb01 pts/1 00:00:00 vi elasticsearch.yml
esuser 3856 1 21 01:15 pts/0 00:00:18 /usr/local/elasticsearch-7.5.1/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=COMPAT -Xms128m -Xmx128m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/elasticsearch-9874251960424438570 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=67108864 -Des.path.home=/usr/local/elasticsearch-7.5.1 -Des.path.conf=/usr/local/elasticsearch-7.5.1/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /usr/local/elasticsearch-7.5.1/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
esuser 3871 3856 0 01:15 pts/0 00:00:00 /usr/local/elasticsearch-7.5.1/modules/x-pack-ml/platform/linux-x86_64/bin/controller
esuser 3921 3521 0 01:16 pts/0 00:00:00 grep --color=auto elasticsearch
[esuser@localhost bin]$ jps
3856 Elasticsearch
3922 Jps
[esuser@localhost bin]$ kill 3856
[esuser@localhost bin]$ jps
3940 Jps
[esuser@localhost bin]$
-结束-