Refactor Faiss-based vector format for easier backport

[kaivalnp]

Description

Refactor classes of the Faiss-based vector format to simplify backport to 10.x

• Extract minimal functionality required for the format into a new FaissLibrary interface
• Separate all function calls to the native library into a new FaissNativeWrapper (and use invokeExact for faster calls!)
• Create a newFaissLibraryNativeImpl class implementing FaissLibrary with FaissNativeWrapper under the hood
• Dynamically load FaissLibraryNativeImpl from FaissLibrary at runtime
• This setup encapsulates "unsafeness" into FaissNativeWrapper and FaissLibraryNativeImpl (as marked by TODOs in build: enable more of javac 24's lints, fix some issues #14907), which can independently be moved into the java21/ directory for an easier backport (see comments in Backport Faiss-based vector format to 10.x #14843), while the FaissKnnVectors{Format/Reader/Writer} can stay as-is!
• Improve some error handling

[github-actions]

This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.

[github-actions]

This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.

[kaivalnp]

Also ran benchmarks to ensure these changes don't adversely affect performance..

main:

recall  latency(ms)  netCPU  avgCpuCount    nDoc  topK  fanout  maxConn  beamWidth  quantized  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  vec_disk(MB)  vec_RAM(MB)  indexType
 0.934        1.649   1.619        0.982  100000   100      50       64        250         no      8.51      11757.79           10.10             1          637.45       292.969      292.969       HNSW

This PR:

recall  latency(ms)  netCPU  avgCpuCount    nDoc  topK  fanout  maxConn  beamWidth  quantized  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  vec_disk(MB)  vec_RAM(MB)  indexType
 0.934        1.637   1.608        0.982  100000   100      50       64        250         no      8.10      12347.20            9.72             1          637.45       292.969      292.969       HNSW

[mikemccand]

Thank you @kaivalnp -- this looks clean to me -- but @uschindler probably has suggestions still ;)

[mikemccand]

Hmm what does this TODO mean again? Is "vectorized" meaning "SIMD instructions"? This term (vector) is overloaded! https://youtu.be/fVq4_HhBK8Y

Maybe clarify to // TODO: use SIMD Faiss API versions where available or so (if that's what it really means)? I think this is necessary because the Faiss build process will produce specific dynamic library for specific SIMD targets (AVX-512 vs AVX-128 etc.)?

[kaivalnp]

Yes sorry, I meant using SIMD instructions wherever available..

Today, the shared library of Faiss' C API (libfaiss_c.so) is linked to the non-SIMD version of the base library (libfaiss.so) by default, but a user can still "point" to the correct SIMD version by changing its dependencies using:

# patchelf --replace-needed OLD_DEPENDENCY NEW_DEPENDENCY SHARED_LIBRARY
patchelf --replace-needed libfaiss.so libfaiss_{avx2,avx512,sve}.so libfaiss_c.so

However, we'd ideally want to do this automatically (either propose a change to upstream Faiss, or something else from Lucene) -- but I wasn't sure how to do it right now..

I'll update the comment soon!

[uschindler]

Give me some time to check this.

In general I agree with those refactorings, it is much cleaner and exception handling is correct. Also everything which touches native code is encapsulated. This is all strongly needed and should be done without any backporting in mind. It is rwquired to have good code.

I don't think we should backport this to our stable 10.x branch as this won't work with Java 21 and still theres complex work to be used to make this compile with Java 21 using APIJARs duplication and so on. My proposal is to release Lucene 11 this autumn with Java 25 (see mailing list thread) where we have no limitations with preview APIs anymore.