[Remote Vector Index Build] Introduce RemoteNativeIndexBuildStrategy skeleton #2525

jed326 · 2025-02-13T01:24:29Z

Description

First PR for #2465

In order to review changes incrementally, this PR is scoped down to only the following:

Feature Flag
Index Setting
Repository name cluster setting
RemoteIndexBuilder skeleton with repository service and index settings wired through the Codec

I am keeping the vector upload changes in a separate follow-up PR as that will deserve it's own in-depth discussion.

The key part of this PR is that we need to pass a Supplier for KNNVectorValues through to the NativeIndexBuildStrategy as we need to open up multiple InputStreams on multiple KNNVectorValues in order to support uploading blobs in parallel. The actual implementation of this parallel upload will come in the next PR, however I am laying the groundwork for that in this skeleton implementation.

Related Issues

Relates: #2465

Check List

New functionality includes testing.
~~- [ ] New functionality has been documented.~~
~~- [ ] API changes companion pull request created.~~
Commits are signed per the DCO using --signoff.
~~- [ ] Public documentation issue/PR created.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

...main/java/org/opensearch/knn/index/codec/KNN10010Codec/KNN10010PerFieldKnnVectorsFormat.java

...in/java/org/opensearch/knn/index/codec/KNN10010Codec/NativeEngines10010KnnVectorsWriter.java

jed326 · 2025-02-13T04:15:56Z

Thanks @navneet1v , I also agree that a new writer isn't completely necessary as the underlying formats are not changing. Moreover, the remote build should be format agnostic anyways, so I've refactored NativeIndexWriter into an interface to be used either locally or remotely when performing flush or merge.

kotwanikunal

Looks good overall @jed326 .

src/main/java/org/opensearch/knn/index/KNNSettings.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

src/main/java/org/opensearch/knn/common/featureflags/KNNFeatureFlags.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/RemoteNativeIndexWriter.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

src/main/java/org/opensearch/knn/plugin/KNNPlugin.java

...a/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriterFlushTests.java

src/main/java/org/opensearch/knn/common/featureflags/KNNFeatureFlags.java

src/main/java/org/opensearch/knn/index/remote/RemoteIndexBuilder.java

src/main/java/org/opensearch/knn/index/codec/KNNCodecService.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/RemoteNativeIndexWriter.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

...a/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriterFlushTests.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/LocalNativeIndexWriter.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/RemoteNativeIndexWriter.java

jed326 · 2025-02-17T01:08:04Z

@shatejas I've reworked this skeleton in the form of RemoteIndexBuildStrategy implements NativeIndexBuildStrategy. Javadoc is still needed in a few places, but this PR is ready for review on the overall approach. Thanks!

jed326 · 2025-02-18T00:28:28Z

...a/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriterFlushTests.java

                    }
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            });
            final Long expectedTimesGetVectorValuesIsCalled = vectorsPerField.stream().filter(Predicate.not(Map::isEmpty)).count();
            knnVectorValuesFactoryMockedStatic.verify(
-                () -> KNNVectorValuesFactory.getVectorValues(any(VectorDataType.class), any(DocsWithFieldSet.class), any()),
-                times(Math.toIntExact(expectedTimesGetVectorValuesIsCalled) * 2)


In quantization case getVectorValues is called twice, however now we are retrieving the supplier itself via getVectorValuesSupplier, which we only need to do once then we are passing the supplier through the rest of the flow.

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/remote/RemoteIndexBuildStrategy.java

Vikasht34

@shatejas already had made some good suggestions on PR , I see those being addressed. I am good with approach and class hierarchy.

jmazanec15

Can we hide repositories service dependency inside some RemoteIndexBuilder abstraction instead of taking direct dependency on it in codec classes?

Also, I think this should go into a feature branch, especially now that main is a release branch. I think overall structure is good, but its still WIP and doesnt have tests.

src/main/java/org/opensearch/knn/index/codec/nativeindex/model/BuildIndexParams.java

jmazanec15 · 2025-02-18T22:00:40Z

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriter.java

@@ -54,15 +55,29 @@ public class NativeEngines990KnnVectorsWriter extends KnnVectorsWriter {
    private final List<NativeEngineFieldVectorsWriter<?>> fields = new ArrayList<>();
    private boolean finished;
    private final Integer approximateThreshold;
+    private final Supplier<RepositoriesService> repositoriesServiceSupplier;


Instead of taking a hard dependency on the RepositoriesService, can we build an abstraction, like RemoteIndexBuilder, that hides these details from the IndexWriter class?

One approach I previously considered was in NativeEngines990KnnVectorsFormat itself we can do the feature checks and then create an instance of RemoteIndexBuildStrategy to pass to the NativeEngines990KnnVectorsWriter. However, it seemed better to me to not instantiate any of the remote index build related classes unless they were actually needed, so I went with this approach of passing the repositoriesServiceSupplier through to the index writer.

Introduced a NativeIndexBuildStrategyFactory class in the latest revision

src/main/java/org/opensearch/knn/index/KNNSettings.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/remote/RemoteIndexBuildStrategy.java

jed326 · 2025-02-18T22:30:43Z

Thanks @jmazanec15

Also, I think this should go into a feature branch, especially now that main is a release branch. I think overall structure is good, but its still WIP and doesnt have tests.

I do have a feature branch in my fork that I have been developing on (https://github.com/jed326/k-NN/commits/remote-vector-staging-2.19/), but I strongly believe we need to start merging these changes into main incrementally otherwise it feels like we are just kicking the can down the road on these high level discussions.

In terms of testing I will add some coverage to ensure the fallback mechanism is working. I was thinking these weren't needed for now since all the functional testing would come in subsequent PRs that include the functionality itself.

jmazanec15 · 2025-02-18T23:27:56Z

@navneet1v @shatejas @Vikasht34 What do you guys think on main vs. feature branch?

Vikasht34 · 2025-02-18T23:31:34Z

@navneet1v @shatejas @Vikasht34 What do you guys think on main vs. feature branch?

Feature Branch

navneet1v · 2025-02-19T00:00:17Z

@navneet1v @shatejas @Vikasht34 What do you guys think on main vs. feature branch?

if tests are there and code is buildable then we should go with main branch. There is no point in feature branch.

jed326 · 2025-02-19T01:09:47Z

@navneet1v @shatejas @Vikasht34 What do you guys think on main vs. feature branch?

I would strongly prefer to not use a feature branch. I think this PR itself is a prime example of why it's important to get a wide range of opinions on changes like this that include a decent amount of refactoring and I think this PR would not have gotten the same amount of feedback if it were directed at a feature branch.

For testing, in the latest revision I have added some randomization to the base test class to randomly enable/disable the new settings and feature flag to ensure the fallback mechanisms are working correctly.

jmazanec15 · 2025-02-19T01:15:34Z

Okay, lets add some tests and we can develop on main, assuming proper feature flag.

… to accept vector value supplier Signed-off-by: Jay Deng <[email protected]>

jed326 · 2025-02-19T01:38:34Z

Added a new RemoteIndexBuildStrategyTests.java as well to test the fallback mechanism. I think that + the settings updates in the KNNRestTestCase base class for ITs should give us good coverage on this change, but please let me know if there's any other coverage you think is missing, thanks!

jed326 requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, ryanbogan, luyuncheng, shatejas, 0ctopus13prime and Vikasht34 as code owners February 13, 2025 01:24

jed326 mentioned this pull request Feb 11, 2025

[Meta] Remote Vector Index Build Component in OpenSearch Vector Engine #2391

Open

15 tasks

jed326 force-pushed the remote-vector-dev branch from 25981af to 8040bae Compare February 13, 2025 01:32

jed326 added v3.0.0 skip-changelog labels Feb 13, 2025

navneet1v reviewed Feb 13, 2025

View reviewed changes

...main/java/org/opensearch/knn/index/codec/KNN10010Codec/KNN10010PerFieldKnnVectorsFormat.java Outdated Show resolved Hide resolved

navneet1v reviewed Feb 13, 2025

View reviewed changes

...in/java/org/opensearch/knn/index/codec/KNN10010Codec/NativeEngines10010KnnVectorsWriter.java Outdated Show resolved Hide resolved

jed326 force-pushed the remote-vector-dev branch 2 times, most recently from 3f39eba to 739d2ef Compare February 13, 2025 04:14

jed326 force-pushed the remote-vector-dev branch from 739d2ef to 38cb684 Compare February 13, 2025 18:11

kotwanikunal reviewed Feb 13, 2025

View reviewed changes

src/main/java/org/opensearch/knn/index/KNNSettings.java Show resolved Hide resolved

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java Outdated Show resolved Hide resolved

jed326 force-pushed the remote-vector-dev branch from 38cb684 to 105ae13 Compare February 13, 2025 19:08

navneet1v reviewed Feb 14, 2025

View reviewed changes

jed326 force-pushed the remote-vector-dev branch 2 times, most recently from e7f50da to 4f8648c Compare February 14, 2025 06:08

shatejas reviewed Feb 14, 2025

View reviewed changes

jed326 force-pushed the remote-vector-dev branch from a154f4d to 4f8648c Compare February 14, 2025 21:41

jed326 force-pushed the remote-vector-dev branch 2 times, most recently from 0531425 to 7e93b7f Compare February 17, 2025 01:06

jed326 force-pushed the remote-vector-dev branch 2 times, most recently from 7964781 to 0c52e14 Compare February 18, 2025 00:26

jed326 commented Feb 18, 2025

View reviewed changes

jmazanec15 reviewed Feb 18, 2025

View reviewed changes

jed326 force-pushed the remote-vector-dev branch from 0c52e14 to 99122fc Compare February 18, 2025 16:45

Vikasht34 previously approved these changes Feb 18, 2025

View reviewed changes

jed326 dismissed Vikasht34’s stale review via cec2c22 February 18, 2025 18:49

jed326 force-pushed the remote-vector-dev branch from 99122fc to cec2c22 Compare February 18, 2025 18:49

jmazanec15 reviewed Feb 18, 2025

View reviewed changes

jed326 force-pushed the remote-vector-dev branch from cec2c22 to e3e057b Compare February 18, 2025 22:57

jed326 force-pushed the remote-vector-dev branch 2 times, most recently from 13b309b to c4a71b6 Compare February 19, 2025 01:05

jed326 force-pushed the remote-vector-dev branch from c4a71b6 to 0ff2432 Compare February 19, 2025 01:19

Introduce RemoteIndexBuildStrategy, refactor NativeIndexBuildStrategy…

b0aa2b4

… to accept vector value supplier Signed-off-by: Jay Deng <[email protected]>

jed326 force-pushed the remote-vector-dev branch from 0ff2432 to b0aa2b4 Compare February 19, 2025 01:37

jed326 changed the title ~~[Remote Vector Index Build] Introduce RemoteIndexBuilder skeleton~~ [Remote Vector Index Build] Introduce RemoteNativeIndexBuildStrategy skeleton Feb 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Vector Index Build] Introduce RemoteNativeIndexBuildStrategy skeleton #2525

[Remote Vector Index Build] Introduce RemoteNativeIndexBuildStrategy skeleton #2525

jed326 commented Feb 13, 2025 •

edited

Loading

jed326 commented Feb 13, 2025

kotwanikunal left a comment

jed326 commented Feb 17, 2025

jed326 Feb 18, 2025

Vikasht34 left a comment

jmazanec15 left a comment

jmazanec15 Feb 18, 2025

jed326 Feb 18, 2025

jed326 Feb 19, 2025

jed326 commented Feb 18, 2025

jmazanec15 commented Feb 18, 2025

Vikasht34 commented Feb 18, 2025

navneet1v commented Feb 19, 2025

jed326 commented Feb 19, 2025

jmazanec15 commented Feb 19, 2025

jed326 commented Feb 19, 2025 •

edited

Loading

[Remote Vector Index Build] Introduce RemoteNativeIndexBuildStrategy skeleton #2525

Are you sure you want to change the base?

[Remote Vector Index Build] Introduce RemoteNativeIndexBuildStrategy skeleton #2525

Conversation

jed326 commented Feb 13, 2025 • edited Loading

Description

Related Issues

Check List

jed326 commented Feb 13, 2025

kotwanikunal left a comment

Choose a reason for hiding this comment

jed326 commented Feb 17, 2025

jed326 Feb 18, 2025

Choose a reason for hiding this comment

Vikasht34 left a comment

Choose a reason for hiding this comment

jmazanec15 left a comment

Choose a reason for hiding this comment

jmazanec15 Feb 18, 2025

Choose a reason for hiding this comment

jed326 Feb 18, 2025

Choose a reason for hiding this comment

jed326 Feb 19, 2025

Choose a reason for hiding this comment

jed326 commented Feb 18, 2025

jmazanec15 commented Feb 18, 2025

Vikasht34 commented Feb 18, 2025

navneet1v commented Feb 19, 2025

jed326 commented Feb 19, 2025

jmazanec15 commented Feb 19, 2025

jed326 commented Feb 19, 2025 • edited Loading

jed326 commented Feb 13, 2025 •

edited

Loading

jed326 commented Feb 19, 2025 •

edited

Loading