Switch to a fixed CFS threshold

[jpountz]

By default, Lucene currently uses compound files for flushed segments, and merged segments that use less than 10% of the total index size (computed either as a number of docs, or as a byte size depending on the merge policy).

I am considering switching to a fixed threshold, e.g. using compound files for all segments below 64MB for byte-size-based merge policies (TieredMergePolicy, LogByteSizeMergePolicy) or 65,536 docs for doc-based merge policies (LogDocMergePolicy).

I like it better for a few reasons:

• Whether a segment is compound or not is more deterministic (and thus easier to reason about) as it doesn't depend on the total size of the index at the time of merging.
• The current ratio doesn't work well in multi-tenant scenarios where you could still have plenty of small files overall due to many small indexes.

Ideally we would have a single switch on IndexWriterConfig instead of having flushes and merges independently make decisions about whether a segments qualifies for being compound.

I'm also wondering if we need to keep the current approach that is based on a ratio, or if only supporting a fixed threshold would be good enough.

[shubhamsrkdev]

@jpountz I was looking at this, it makes sense but I had a question about the last part :

I'm also wondering if we need to keep the current approach that is based on a ratio, or if only supporting a fixed threshold would be good enough.

I understand that at this point the CFS ratio is set in merge policies (setNoCFSRatio), if we intend to keep the two things simultaneously, wouldn't it be too confusing? That is, CFS ratio is set in merge polices but the fixed threshold be set in IndexWriterConfig. Also, there would be many more permutations on whether a particular (type?) of CFS threshold is chosen (ratio based or fixed).

[jpountz]

I agree. If there is no strong argument for having both then we should only support setting a fixed CFS threshold?

[jpountz]

Alternate idea: maybe this should be fully controlled by the CompoundFormat. It would allow it to do something like packing the smaller files into a compound file and keeping the bigger files on their own, e.g. sorting files by increasing size and adding them to a compound file as long as the compound file is less than or equal to 16MB.