[Semantic Text] Integration Test

[Samiul-TheSoccerFan]

An integration test to:

• Select a set of index versions randomly from the available versions.
• Creates an index for each selected index version
• Ingests data
• Performs semantic searches (including highlighting when applicable)

[elasticsearchmachine]

Pinging @elastic/search-eng (Team:SearchOrg)

[elasticsearchmachine]

Pinging @elastic/search-relevance (Team:Search - Relevance)

[Samiul-TheSoccerFan]

@elasticmachine update branch

[Samiul-TheSoccerFan]

@elasticmachine update branch

[Samiul-TheSoccerFan]

This index version starts from 8.15. If we use the SEMANTIC_TEXT_FIELD_TYPE, the index version starts from 8.14.4/5 and fails to create the semantic_text field.

[Mikep86]

I just tried starting from SEMANTIC_TEXT_FIELD_TYPE and the test passes. Perhaps the test failure was related to other problems that you've now fixed?

[Mikep86]

Good start to this, I left some comments on how to move it forward

[kderusso]

Interesting, we should follow up on this. It's strange that no plugin registers SparseVectorQueryBuilder as a named writeable, yet your test requires it.

@Mikep86 @Samiul-TheSoccerFan the query is registered in XPackClientPlugin did you try pulling it from there?

[Mikep86]

@kderusso

the query is registered in XPackClientPlugin did you try pulling it from there?

XPackClientPlugin only registers it as a query, not a named writeable. It shouldn't be necessary to register it as a named writeable because rewriting to sparse_vector happens on the shard.

We debugged the issue a bit and the serialization happens when generating the search response in the test, after the query execution is complete. Adding to the mystery, this serialization does not happen when executing a semantic query in production. So the test is following a different code path that we need to understand.

[Mikep86]

@Samiul-TheSoccerFan @kderusso Figured out what's going on with SparseVectorQueryBuilder named writeable. The query goes through a serialization/deserialization cycle as part of creating the search response (both in prod and test code), but only in a multi-node cluster. The test fails when it (randomly) uses a multi-node cluster because SparseVectorQueryBuilder is not registered as a named writeable. But why not?

Queries exposed via getQueries are registered as queries and named writeables. SparseVectorQueryBuilder is registered as a query by XPackClientPlugin. LocalStateInferencePlugin extends XPackClientPlugin, but we have overridden getQueries, preventing SparseVectorQueryBuilder from being registered. It doesn't work to add XPackClientPlugin as a separate plugin dependency since that would result in double inclusion of elements that aren't overridden by LocalStateInferencePlugin (it's also a messier solution). Instead, the correct solution is to ensure that LocalStateInferencePlugin.getQueries registers SparseVectorQueryBuilder.

IMO the way to do this is to add the following to LocalStateCompositeXPackPlugin:

    @Override
    public List<QuerySpec<?>> getQueries() {
        List<QuerySpec<?>> querySpecs = new ArrayList<>(super.getQueries()); // The xpack plugin also registers queries, include those
        filterPlugins(SearchPlugin.class).stream().flatMap(p -> p.getQueries().stream()).forEach(querySpecs::add);

        return querySpecs;
    }

This should handle the case for every plugin that extends LocalStateCompositeXPackPlugin, saving us some future headaches.

[Samiul-TheSoccerFan]

@elasticmachine update branch

[Mikep86]

Looking much better, two small things to adjust and I think we're good :)

[Mikep86]

I just tried starting from SEMANTIC_TEXT_FIELD_TYPE and the test passes. Perhaps the test failure was related to other problems that you've now fixed?

[Samiul-TheSoccerFan]

@elasticmachine update branch

[Mikep86]

@elasticmachine update branch

[Samiul-TheSoccerFan]

@elasticmachine update branch

[kderusso]

LGTM - Very nice work, and thanks for all of the iterations! 👏

[Mikep86]

I was waiting for green CI, but it looks like that's hosed for unrelated reasons. Good work!

[Samiul-TheSoccerFan]

@elasticmachine update branch

[Samiul-TheSoccerFan]

@elasticmachine update branch

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x