Skip to content

Commit ba455f2

Browse files
Pulkitg64Pulkit Gupta
andauthored
Add fallback support to Lucene104ScalarQuantizedVectorsFormat getFloatVectorValues when there are no full-precision vectors present (#15415)
* Add fallback support to Lucene104ScalarQuantizedVectorsFormat.getFloatVectorValues when there are no full-precision vectors present * Removed some unused params and tidy changes * Added entry in Changes.txt * Added name in Changes.txt log * Addressed comments * Renamed function params --------- Co-authored-by: Pulkit Gupta <[email protected]>
1 parent e02bdb4 commit ba455f2

File tree

6 files changed

+654
-20
lines changed

6 files changed

+654
-20
lines changed

lucene/CHANGES.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,9 @@ New Features
176176
`Lucene104HnswScalarQuantizedVectorsFormat(ScalarEncoding.SINGLE_BIT_QUERY_NIBBLE, int, int)`
177177
(Ben Trent)
178178

179+
* GITHUB#15415: Add fallback support to Lucene104ScalarQuantizedVectorsFormat getFloatVectorValues when there are
180+
no full-precision vectors present (Pulkit Gupta)
181+
179182
Improvements
180183
---------------------
181184
* GITHUB#15148: Add support uint8 distance and allow 8 bit scalar quantization (Trevor McCulloch)

lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsReader.java

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,23 @@ public FloatVectorValues getFloatVectorValues(String field) throws IOException {
210210
+ " expected: "
211211
+ VectorEncoding.FLOAT32);
212212
}
213+
214+
FloatVectorValues rawFloatVectorValues = rawVectorsReader.getFloatVectorValues(field);
215+
216+
if (rawFloatVectorValues.size() == 0) {
217+
return OffHeapScalarQuantizedFloatVectorValues.load(
218+
fi.ordToDocDISIReaderConfiguration,
219+
fi.dimension,
220+
fi.size,
221+
fi.scalarEncoding,
222+
fi.similarityFunction,
223+
vectorScorer,
224+
fi.centroid,
225+
fi.vectorDataOffset,
226+
fi.vectorDataLength,
227+
quantizedVectorData);
228+
}
229+
213230
OffHeapScalarQuantizedVectorValues sqvv =
214231
OffHeapScalarQuantizedVectorValues.load(
215232
fi.ordToDocDISIReaderConfiguration,
@@ -224,7 +241,7 @@ public FloatVectorValues getFloatVectorValues(String field) throws IOException {
224241
fi.vectorDataOffset,
225242
fi.vectorDataLength,
226243
quantizedVectorData);
227-
return new ScalarQuantizedVectorValues(rawVectorsReader.getFloatVectorValues(field), sqvv);
244+
return new ScalarQuantizedVectorValues(rawFloatVectorValues, sqvv);
228245
}
229246

230247
@Override

0 commit comments

Comments
 (0)