我们都知道 elastic search 是近实时的搜索系统,这里面的原因究竟是什么呢?es 是近实时,是因为 lucene 是近实时的。我们看一段 lucene 的示例代码:
使用的 lucene 版本如下:
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>8.9.0</version>
</dependency>
1 Analyzer analyzer = new StandardAnalyzer();
2
3 Path indexPath = new File("tmp_index").toPath();
4 IOUtils.rm(indexPath);
5 Directory directory = FSDirectory.open(indexPath);
6 IndexWriterConfig config = new IndexWriterConfig(analyzer);
7 // 使用 compound file
8 config.setUseCompoundFile(true);
9 // config.setInfoStream(System.out);
10 IndexWriter iw = new IndexWriter(directory, config);
11 // Now search the index:
12 StandardDirectoryReader oldReader = (StandardDirectoryReader) DirectoryReader.open(iw);
13
14 // 添加文档 1
15 Document doc = new Document();
16 doc.add(new Field("name", "Pride and Prejudice", TextField.TYPE_STORED));
17 doc.add(new Field("line", "It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife.", TextField.TYPE_STORED));
18 doc.add(new Field("chapters", "1", TextField.TYPE_STORED));
19 iw.addDocument(doc);
20
21 IndexSearcher searcher = new IndexSearcher(oldReader);
22 // Parse a simple query that searches for "text":
23 QueryParser parser = new QueryParser("name", analyzer);
24 Query query = parser.parse("prejudice");
25 ScoreDoc[] hits = searcher.search(query, 10).scoreDocs;
26 // Iterate through the results:
27 System.out.println("第一次查询:");
28 System.out.printf("version=%d, segmentInfos=%s\n", oldReader.getVersion(),
29 oldReader.getSegmentInfos());
30 for (int i = 0; i < hits.length; i++) {
31 System.out.println(hits[i]);
32 }
33
34 StandardDirectoryReader newReader = (StandardDirectoryReader) DirectoryReader.openIfChanged(oldReader);
35 oldReader.close();
36
37 searcher = new IndexSearcher(newReader);
38 hits = searcher.search(query, 10).scoreDocs;
39 System.out.println("第二次查询:");
40 System.out.printf("version=%d, segmentInfos=%s\n", newReader.getVersion(),
41 newReader.getSegmentInfos());
42 // Iterate through the results:
43 for (int i = 0; i < hits.length; i++) {
44 System.out.println(hits[i]);
45 }
46
47 iw.commit();
48 newReader.close();
49 directory.close();
在第 15 行处,添加一个文档,然后执行搜索,并没有查询到文档。
在第 34 行处,调用 DirectoryReader.openIfChanged 方法,然后使用新的 reader 执行搜索,则可以查询到文档,注意此时文档仍然没有 flush 到磁盘中。
在第 47 行处,调用 IndexWriter.commit 方法,执行 flush 写盘。
OK ,现在看到了近实时的现象,对应到 ES 中的操作,在 ES 中写入一个文档,如同 15 行的操作,写入并不是立即可见;
需要执行 refresh,对应到第 24 行代码的 openIfChanged 方法;
ES 执行 flush 则对应到第 47 行的 IndexWriter.commit 方法。
针对网上所说的,ES 每次 refresh 之后都会生成一个 segment,看看 debug 截图:
openIfChanged 之后确实生成了一个 SegmentReader
附加2个小问题:1. 当不指定文档ID时,这个ID是如何生成的?ES确实有一个算法
// org.elasticsearch.common.TimeBasedUUIDGenerator#getBase64UUID
public String getBase64UUID() {
final int sequenceId = sequenceNumber.incrementAndGet() & 0xffffff;
long currentTimeMillis = currentTimeMillis();
long timestamp = this.lastTimestamp.updateAndGet(lastTimestamp -> {
// Don't let timestamp go backwards, at least "on our watch" (while this JVM is running). We are
// still vulnerable if we are shut down, clock goes backwards, and we restart... for this we
// randomize the sequenceNumber on init to decrease chance of collision:
long nonBackwardsTimestamp = Math.max(lastTimestamp, currentTimeMillis);
if (sequenceId == 0) {
// Always force the clock to increment whenever sequence number is 0, in case we have a long
// time-slip backwards:
nonBackwardsTimestamp++;
}
return nonBackwardsTimestamp;
});
final byte[] uuidBytes = new byte[15];
int i = 0;
// We have auto-generated ids, which are usually used for append-only workloads.
// So we try to optimize the order of bytes for indexing speed (by having quite
// unique bytes close to the beginning of the ids so that sorting is fast) and
// compression (by making sure we share common prefixes between enough ids),
// but not necessarily for lookup speed (by having the leading bytes identify
// segments whenever possible)
// Blocks in the block tree have between 25 and 48 terms. So all prefixes that
// are shared by ~30 terms should be well compressed. I first tried putting the
// two lower bytes of the sequence id in the beginning of the id, but compression
// is only triggered when you have at least 30*2^16 ~= 2M documents in a segment,
// which is already quite large. So instead, we are putting the 1st and 3rd byte
// of the sequence number so that compression starts to be triggered with smaller
// segment sizes and still gives pretty good indexing speed. We use the sequenceId
// rather than the timestamp because the distribution of the timestamp depends too
// much on the indexing rate, so it is less reliable.
uuidBytes[i++] = (byte) sequenceId;
// changes every 65k docs, so potentially every second if you have a steady indexing rate
uuidBytes[i++] = (byte) (sequenceId >>> 16);
// Now we start focusing on compression and put bytes that should not change too often.
uuidBytes[i++] = (byte) (timestamp >>> 16); // changes every ~65 secs
uuidBytes[i++] = (byte) (timestamp >>> 24); // changes every ~4.5h
uuidBytes[i++] = (byte) (timestamp >>> 32); // changes every ~50 days
uuidBytes[i++] = (byte) (timestamp >>> 40); // changes every 35 years
byte[] macAddress = macAddress();
assert macAddress.length == 6;
System.arraycopy(macAddress, 0, uuidBytes, i, macAddress.length);
i += macAddress.length;
// Finally we put the remaining bytes, which will likely not be compressed at all.
uuidBytes[i++] = (byte) (timestamp >>> 8);
uuidBytes[i++] = (byte) (sequenceId >>> 8);
uuidBytes[i++] = (byte) timestamp;
assert i == uuidBytes.length;
return Base64.getUrlEncoder().withoutPadding().encodeToString(uuidBytes);
}
2. 网上传言当 index buffer 满时,会触发 refresh 的动作,判断 buffer 是否满,这个参数到底是啥
/**
* Determines the amount of RAM that may be used for buffering added documents
* and deletions before they are flushed to the Directory. Generally for
* faster indexing performance it's best to flush by RAM usage instead of
* document count and use as large a RAM buffer as you can.
* <p>
* When this is set, the writer will flush whenever buffered documents and
* deletions use this much RAM. Pass in
* {@link IndexWriterConfig#DISABLE_AUTO_FLUSH} to prevent triggering a flush
* due to RAM usage. Note that if flushing by document count is also enabled,
* then the flush will be triggered by whichever comes first.
* <p>
* The maximum RAM limit is inherently determined by the JVMs available
* memory. Yet, an {@link IndexWriter} session can consume a significantly
* larger amount of memory than the given RAM limit since this limit is just
* an indicator when to flush memory resident documents to the Directory.
* Flushes are likely happen concurrently while other threads adding documents
* to the writer. For application stability the available memory in the JVM
* should be significantly larger than the RAM buffer used for indexing.
* <p>
* <b>NOTE</b>: the account of RAM usage for pending deletions is only
* approximate. Specifically, if you delete by Query, Lucene currently has no
* way to measure the RAM usage of individual Queries so the accounting will
* under-estimate and you should compensate by either calling commit() or refresh()
* periodically yourself.
* <p>
* <b>NOTE</b>: It's not guaranteed that all memory resident documents are
* flushed once this limit is exceeded. Depending on the configured
* {@link FlushPolicy} only a subset of the buffered documents are flushed and
* therefore only parts of the RAM buffer is released.
* <p>
*
* The default value is {@link IndexWriterConfig#DEFAULT_RAM_BUFFER_SIZE_MB}.
*
* <p>
* Takes effect immediately, but only the next time a document is added,
* updated or deleted.
*
* @see IndexWriterConfig#setRAMPerThreadHardLimitMB(int)
*
* @throws IllegalArgumentException
* if ramBufferSize is enabled but non-positive, or it disables
* ramBufferSize when maxBufferedDocs is already disabled
*/
public synchronized LiveIndexWriterConfig setRAMBufferSizeMB(double ramBufferSizeMB) {
if (ramBufferSizeMB != IndexWriterConfig.DISABLE_AUTO_FLUSH && ramBufferSizeMB <= 0.0) {
throw new IllegalArgumentException("ramBufferSize should be > 0.0 MB when enabled");
}
if (ramBufferSizeMB == IndexWriterConfig.DISABLE_AUTO_FLUSH
&& maxBufferedDocs == IndexWriterConfig.DISABLE_AUTO_FLUSH) {
throw new IllegalArgumentException("at least one of ramBufferSize and maxBufferedDocs must be enabled");
}
this.ramBufferSizeMB = ramBufferSizeMB;
return this;
}
该参数默认是 16 MB
// org.apache.lucene.index.IndexWriterConfig
public static final double DEFAULT_RAM_BUFFER_SIZE_MB = 16.0;