1 搜索示例

首先在lucene索引中预先写入了一些文档，主要包含两个field (id和name)信息，每个field都是stored和indexed

{
    "id": 0,
    "name": "Stephen"
},{
    "id": 1,
    "name": "Draymond"
},{
    "id": 2,
    "name", "LeBron"
},{
    "id": 3,
    "name": "Kevin"
}

我们使用如下代码，可以从lucene索引中搜索name为LeBron的文档信息，接下来，我们将通过一系列文章来分析这些代码是如何工作的

public class IndexSearcherTest {
    public static void main(String[] args) throws IOException, ParseException {
        Directory directory = FSDirectory.open("/lucene/index/path");
        IndexReader indexReader = DirectoryReader.open(directory);

        IndexSearcher indexSearcher = new IndexSearcher(indexReader);
        
        QueryParser queryParser = new QueryParser("name", new StandardAnalyzer());
        Query query = queryParser.parse("LeBron");

        TopDocs topDocs = indexSearcher.search(query, 10);

        ScoreDoc[] scoreDocs = topDocs.scoreDocs;
        for(ScoreDoc scoreDoc : scoreDocs){
            System.out.println("doc: " + scoreDoc.doc + ", score: " + scoreDoc.score);

            Document document = indexReader.document(scoreDoc.doc);
            if(null == document){
                continue;
            }
            System.out.println("id :" + document.get("id") + ", name: " + document.get("name"));
        }
    }
}

本片文章主要介绍打开索引路径和索引目录的部分

1 打开索引文件

由于索引存在本地磁盘中，可以使用FSDirectory打开本地的索引文件，获取索引路径的Directory对象

Directory directory = FSDirectory.open("/lucene/index/path");

① 如果当前jre是64位且支持Unmap（能加载sun.misc.Cleaner类和java.nio.DirectByteBuffer.cleaner()方法），则创建的是MMapDirectory对象
② 如果当前系统是Windows（判断操作系统名称是否以Windows开头），则创建SimpleFSDirectory对象
③ 如果不满足上述两种情况，则创建NIOFSDirectory对象

public abstract class FSDirectory extends BaseDirectory {
  public static FSDirectory open(File path) throws IOException {
    return open(path, null);
  }

  public static FSDirectory open(File path, LockFactory lockFactory) throws IOException {
    if (Constants.JRE_IS_64BIT && MMapDirectory.UNMAP_SUPPORTED) {
      return new MMapDirectory(path, lockFactory);
    } else if (Constants.WINDOWS) {
      return new SimpleFSDirectory(path, lockFactory);
    } else {
      return new NIOFSDirectory(path, lockFactory);
    }
  }
}

上述三种Directory的构造方法都是通过调用父类FSDirectory实现的

FSDirectory类图

在FSDirectory的构造方法中，主要是初始化了lockFactory和directory对象

public abstract class FSDirectory extends BaseDirectory {
  protected FSDirectory(File path, LockFactory lockFactory) throws IOException {
    // new ctors use always NativeFSLockFactory as default:
    if (lockFactory == null) {
      lockFactory = new NativeFSLockFactory();
    }
    directory = path.getCanonicalFile();

    if (directory.exists() && !directory.isDirectory())
      throw new NoSuchDirectoryException("file '" + directory + "' exists but is not a directory");

    setLockFactory(lockFactory);

  }
}

2 打开索引目录

2.1 查找segment文件

在打开索引路径获得索引目录后，会使用DirectoryReader打开这个索引目录

IndexReader indexReader = DirectoryReader.open(directory);

这个过程非常复杂，主要是查找并打开索引的segment文件（segments_N和segments.gen），从segment中获取index文件信息并打开（.tip， .tim，.doc，.pos，.tvd，.tvx，.si，.nvd，.nvm）

public abstract class DirectoryReader extends BaseCompositeReader<AtomicReader> {
  public static DirectoryReader open(final Directory directory) throws IOException {
    return StandardDirectoryReader.open(directory, null, DEFAULT_TERMS_INDEX_DIVISOR);
  }
}

在StandardDirectoryReader的open()方法中，主要是创建SegmentInfos.FindSegmentsFile对象并重写doBody()方法，然后执行对象的run()方法

final class StandardDirectoryReader extends DirectoryReader {
  static DirectoryReader open(final Directory directory, final IndexCommit commit,
                          final int termInfosIndexDivisor) throws IOException {
    return (DirectoryReader) new SegmentInfos.FindSegmentsFile(directory) {
      @Override
      protected Object doBody(String segmentFileName) throws IOException {
        SegmentInfos sis = new SegmentInfos();
        sis.read(directory, segmentFileName);
        final SegmentReader[] readers = new SegmentReader[sis.size()];
        boolean success = false;
        try {
          for (int i = sis.size()-1; i >= 0; i--) {
            readers[i] = new SegmentReader(sis.info(i), termInfosIndexDivisor, IOContext.READ);
          }

          DirectoryReader reader = new StandardDirectoryReader(directory, readers, null, sis, termInfosIndexDivisor, false);
          success = true;

          return reader;
        } finally {
          if (success == false) {
            IOUtils.closeWhileHandlingException(readers);
          }
        }
      }
    }.run(commit);
  }
}

SegmentInfos.FindSegmentsFile的run()方法主要是查找segment文件
① 从以segment开头但不为segments.gen的文件中查找后缀最大的字符的作为genA
② 读取segment.gen文件，从lucene索引文件格式可知其格式如下：

GenHeader	Generation	Generation	Footer

generation为一个Long类型的数字，并且被写入了两次，如果两个值相同，则作为genB
③ 比较genA和genB的值，最大的作为最终的gen值，用segment_[gen]作为segment文件名

public final class SegmentInfos implements Cloneable, Iterable<SegmentCommitInfo> {
  public abstract static class FindSegmentsFile {
    public Object run(IndexCommit commit) throws IOException {
      if (commit != null) {
        if (directory != commit.getDirectory())
          throw new IOException("the specified commit does not match the specified Directory");
        return doBody(commit.getSegmentsFileName());
      }

      String segmentFileName = null;
      long gen = 0;
      int retryCount = 0;

      boolean useFirstMethod = true;

      while(true) {
        if (useFirstMethod) {
          files = directory.listAll();
          
          if (files != null) {
            genA = getLastCommitGeneration(files);
          }
          
          long genB = -1;
          try {
            genInput = directory.openChecksumInput(IndexFileNames.SEGMENTS_GEN, IOContext.READONCE);
          } catch (IOException e) {
          
          }
  
          if (genInput != null) {
            try {
              int version = genInput.readInt();
              if (version == FORMAT_SEGMENTS_GEN_47 || version == FORMAT_SEGMENTS_GEN_CHECKSUM) {
                long gen0 = genInput.readLong();
                long gen1 = genInput.readLong();
                if (gen0 == gen1) {
                  // The file is consistent.
                  genB = gen0;
                }
              } else {
                throw new IndexFormatTooNewException(genInput, version, FORMAT_SEGMENTS_GEN_START, FORMAT_SEGMENTS_GEN_CURRENT);
              }
            } catch (IOException err2) {
             
          }

          gen = Math.max(genA, genB);

        if (useFirstMethod && lastGen == gen && retryCount >= 2) {
          useFirstMethod = false;
        }

        segmentFileName = IndexFileNames.fileNameFromGeneration(IndexFileNames.SEGMENTS,
                                                                "",
                                                                gen);
        try {
          Object v = doBody(segmentFileName);
          if (infoStream != null) {
            message("success on " + segmentFileName);
          }
          return v;
        } catch (IOException err) {
          // ...
        }
      }
    }
    protected abstract Object doBody(String segmentFileName) throws IOException;
  }
}

2.2 打开index文件

在SegmentInfos.FindSegmentsFile的doBody()方法中，主要是读取各个index文件

final class StandardDirectoryReader extends DirectoryReader {
  static DirectoryReader open(final Directory directory, final IndexCommit commit,
                          final int termInfosIndexDivisor) throws IOException {
    return (DirectoryReader) new SegmentInfos.FindSegmentsFile(directory) {
      @Override
      protected Object doBody(String segmentFileName) throws IOException {
        SegmentInfos sis = new SegmentInfos();
        sis.read(directory, segmentFileName);
        final SegmentReader[] readers = new SegmentReader[sis.size()];
        boolean success = false;
        try {
          for (int i = sis.size()-1; i >= 0; i--) {
            readers[i] = new SegmentReader(sis.info(i), termInfosIndexDivisor, IOContext.READ);
          }

          // This may throw IllegalArgumentException if there are too many docs, so
          // it must be inside try clause so we close readers in that case:
          DirectoryReader reader = new StandardDirectoryReader(directory, readers, null, sis, termInfosIndexDivisor, false);
          success = true;

          return reader;
        } finally {
          if (success == false) {
            IOUtils.closeWhileHandlingException(readers);
          }
        }
      }
    }.run(commit);
  }
}

首先创建一个SegmentInfos()对象，然后调用sis.read(directory, segmentFileName);读取segment文件
segment文件格式如下：

Header	Version	NameCounter	SegCount	SegCount	CommitUserData	Footer

然后遍历segment中的每一个段信息，调用SegmentReader的构造方法读取索引的segment 索引文件

public final class SegmentReader extends AtomicReader implements Accountable {
  public SegmentReader(SegmentCommitInfo si, int termInfosIndexDivisor, IOContext context) throws IOException {
    this.si = si;
    // 读取cfs
    fieldInfos = readFieldInfos(si);
    // 读取tip tim nvd nvm fdt fdx tvf tvd tvx
    core = new SegmentCoreReaders(this, si.info.dir, si, context, termInfosIndexDivisor);
    segDocValues = new SegmentDocValues();
    
    boolean success = false;
    final Codec codec = si.info.getCodec();
    try {
      if (si.hasDeletions()) {
        读取 del
        liveDocs = codec.liveDocsFormat().readLiveDocs(directory(), si, IOContext.READONCE);
      } else {
        assert si.getDelCount() == 0;
        liveDocs = null;
      }
      numDocs = si.info.getDocCount() - si.getDelCount();

      if (fieldInfos.hasDocValues()) {
        initDocValuesProducers(codec);
      }

      success = true;
    } finally {
      if (!success) {
        doClose();
      }
    }
  }
}

① 用readFieldInfos()方法读取cfs文件，一个“虚拟”的文件，用于访问复合流

public final class SegmentReader extends AtomicReader implements Accountable {
  static FieldInfos readFieldInfos(SegmentCommitInfo info) throws IOException {
    final Directory dir;
    final boolean closeDir;
    if (info.getFieldInfosGen() == -1 && info.info.getUseCompoundFile()) {
      // no fieldInfos gen and segment uses a compound file
      dir = new CompoundFileDirectory(info.info.dir,
          IndexFileNames.segmentFileName(info.info.name, "", IndexFileNames.COMPOUND_FILE_EXTENSION),
          IOContext.READONCE,
          false);
      closeDir = true;
    } else {
      // gen'd FIS are read outside CFS, or the segment doesn't use a compound file
      dir = info.info.dir;
      closeDir = false;
    }
    
    try {
      final String segmentSuffix = info.getFieldInfosGen() == -1 ? "" : Long.toString(info.getFieldInfosGen(), Character.MAX_RADIX);
      Codec codec = info.info.getCodec();
      FieldInfosFormat fisFormat = codec.fieldInfosFormat();
      return fisFormat.getFieldInfosReader().read(dir, info.info.name, segmentSuffix, IOContext.READONCE);
    } finally {
      if (closeDir) {
        dir.close();
      }
    }
  }
}

② 构建SegmentCoreReaders对象时主要读取tip tim nvd nvm fdt fdx tvf tvd tvx 格式文件

final class SegmentCoreReaders implements Accountable {
  SegmentCoreReaders(SegmentReader owner, Directory dir, SegmentCommitInfo si, IOContext context, int termsIndexDivisor) throws IOException {

    if (termsIndexDivisor == 0) {
      throw new IllegalArgumentException("indexDivisor must be < 0 (don't load terms index) or greater than 0 (got 0)");
    }
    
    final Codec codec = si.info.getCodec();
    final Directory cfsDir; // confusing name: if (cfs) its the cfsdir, otherwise its the segment's directory.

    boolean success = false;
    
    try {
      if (si.info.getUseCompoundFile()) {
        // 读取cfs 文件
        cfsDir = cfsReader = new CompoundFileDirectory(dir, IndexFileNames.segmentFileName(si.info.name, "", IndexFileNames.COMPOUND_FILE_EXTENSION), context, false);
      } else {
        cfsReader = null;
        cfsDir = dir;
      }

      final FieldInfos fieldInfos = owner.fieldInfos;
      
      this.termsIndexDivisor = termsIndexDivisor;
      final PostingsFormat format = codec.postingsFormat();
      final SegmentReadState segmentReadState = new SegmentReadState(cfsDir, si.info, fieldInfos, context, termsIndexDivisor);
      // 读取tip 和 tim 文件
      fields = format.fieldsProducer(segmentReadState);
      assert fields != null;

      if (fieldInfos.hasNorms()) {
        // 读取 nvd和 nvm 文件
        normsProducer = codec.normsFormat().normsProducer(segmentReadState);
        assert normsProducer != null;
      } else {
        normsProducer = null;
      }
      // 读取fdx 和fdt 文件
      fieldsReaderOrig = si.info.getCodec().storedFieldsFormat().fieldsReader(cfsDir, si.info, fieldInfos, context);
      if (fieldInfos.hasVectors()) { 
        // 读取 tvf  tvd  和  tvx 文件
        termVectorsReaderOrig = si.info.getCodec().termVectorsFormat().vectorsReader(cfsDir, si.info, fieldInfos, context);
      } else {
        termVectorsReaderOrig = null;
      }

      success = true;
    } finally {
      if (!success) {
        decRef();
      }
    }
  }
}

tim文件格式如下：

Header	FSTIndexNumFields	<IndexStartFP>NumFields	DirOffset	Footer

tip文件格式如下：

Header	PostingsHeader	NumBlocks	FieldSummary	DirOffset	Footer

public class BlockTreeTermsReader extends FieldsProducer {
  public BlockTreeTermsReader(Directory dir, FieldInfos fieldInfos, SegmentInfo info,
                              PostingsReaderBase postingsReader, IOContext ioContext,
                              String segmentSuffix, int indexDivisor)
    throws IOException {
    
    this.postingsReader = postingsReader;

    this.segment = info.name;
    // 读取cfs
    in = dir.openInput(IndexFileNames.segmentFileName(segment, segmentSuffix, BlockTreeTermsWriter.TERMS_EXTENSION),
                       ioContext);

    boolean success = false;
    IndexInput indexIn = null;

    try {
      version = readHeader(in);
      if (indexDivisor != -1) {
        indexIn = dir.openInput(IndexFileNames.segmentFileName(segment, segmentSuffix, BlockTreeTermsWriter.TERMS_INDEX_EXTENSION),
                                ioContext);
        int indexVersion = readIndexHeader(indexIn);
        if (indexVersion != version) {
          throw new CorruptIndexException("mixmatched version files: " + in + "=" + version + "," + indexIn + "=" + indexVersion);
        }
      }
      
      // verify
      if (indexIn != null && version >= BlockTreeTermsWriter.VERSION_CHECKSUM) {
        CodecUtil.checksumEntireFile(indexIn);
      }

      // Have PostingsReader init itself
      postingsReader.init(in);

      // ...
  }
}

fdx 文件格式

<Header>	<FieldValuesPosition> SegSize

fdt文件格式

<Header>	<DocFieldData> SegSize

public final class Lucene40StoredFieldsReader extends StoredFieldsReader implements Cloneable, Closeable {
  public Lucene40StoredFieldsReader(Directory d, SegmentInfo si, FieldInfos fn, IOContext context) throws IOException {
    final String segment = si.name;
    boolean success = false;
    fieldInfos = fn;
    try {
      fieldsStream = d.openInput(IndexFileNames.segmentFileName(segment, "", FIELDS_EXTENSION), context);
      final String indexStreamFN = IndexFileNames.segmentFileName(segment, "", FIELDS_INDEX_EXTENSION);
      indexStream = d.openInput(indexStreamFN, context);
      
      CodecUtil.checkHeader(indexStream, CODEC_NAME_IDX, VERSION_START, VERSION_CURRENT);
      CodecUtil.checkHeader(fieldsStream, CODEC_NAME_DAT, VERSION_START, VERSION_CURRENT);
      assert HEADER_LENGTH_DAT == fieldsStream.getFilePointer();
      assert HEADER_LENGTH_IDX == indexStream.getFilePointer();
      final long indexSize = indexStream.length() - HEADER_LENGTH_IDX;
      this.size = (int) (indexSize >> 3);
      // Verify two sources of "maxDoc" agree:
      if (this.size != si.getDocCount()) {
        throw new CorruptIndexException("doc counts differ for segment " + segment + ": fieldsReader shows " + this.size + " but segmentInfo shows " + si.getDocCount());
      }
      numTotalDocs = (int) (indexSize >> 3);
      success = true;
    } finally {
      if (!success) {
        try {
          close();
        } catch (Throwable t) {} // ensure we throw our original exception
      }
    }
  }
}

tvx 文件格式

Header	<DocumentPosition,FieldPosition> NumDocs

tvd 文件格式

Header	<NumFields, FieldNums, FieldPositions> NumDocs

tvf 文件格式

Header	<NumTerms, Flags, TermFreqs> NumFields

③ liveDocs = codec.liveDocsFormat().readLiveDocs(directory(), si, IOContext.READONCE);主要读取del文件
del文件格式

Format	Header	ByteCount	BitCount	Bits

public class Lucene40LiveDocsFormat extends LiveDocsFormat {
  public Bits readLiveDocs(Directory dir, SegmentCommitInfo info, IOContext context) throws IOException {
    String filename = IndexFileNames.fileNameFromGeneration(info.info.name, DELETES_EXTENSION, info.getDelGen());
    final BitVector liveDocs = new BitVector(dir, filename, context);
    if (liveDocs.length() != info.info.getDocCount()) {
      throw new CorruptIndexException("liveDocs.length()=" + liveDocs.length() + "info.docCount=" + info.info.getDocCount() + " (filename=" + filename + ")");
    }
    if (liveDocs.count() != info.info.getDocCount() - info.getDelCount()) {
      throw new CorruptIndexException("liveDocs.count()=" + liveDocs.count() + " info.docCount=" + info.info.getDocCount() + " info.getDelCount()=" + info.getDelCount() + " (filename=" + filename + ")");
    }
    return liveDocs;
  }
}

3 创建IndexSearcher

在打开索引目录后，接着创建IndexSearcher对象

IndexSearcher indexSearcher = new IndexSearcher(indexReader);

创建IndexSearcher的过程，主要是初始化searcher的context和reader

public class IndexSearcher {
  public IndexSearcher(IndexReader r) {
    this(r, null);
  }
  public IndexSearcher(IndexReader r, ExecutorService executor) {
    this(r.getContext(), executor);
  }
  public IndexSearcher(IndexReaderContext context, ExecutorService executor) {
    assert context.isTopLevel: "IndexSearcher's ReaderContext must be topLevel for reader" + context.reader();
    reader = context.reader();
    this.executor = executor;
    this.readerContext = context;
    leafContexts = context.leaves();
    this.leafSlices = executor == null ? null : slices(leafContexts);
  }
}