(#93) Update to Elasticsearch 8 #97
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Refactor project to use Elasticsearch 8 - Correct target framework to net8.0 - Remove unused CustomJsonConverter.
…lasticsearch v8 This commit updates the testing library to use the new Elasticsearch client v8.19.* and removes dependencies on NEST v7 and Newtonsoft.Json. File-by-file changes: src/NCI.OCPL.Api.Common.Testing/NCI.OCPL.Api.Common.Testing.csproj: - Replaced NEST (v7.9.*) package with Elastic.Clients.Elasticsearch (v8.19.*) - Removed NEST.JsonNetSerializer (v7.9.*) package - Removed Microsoft.Extensions.Logging.Testing.Library package reference - Updated to use net8.0 target framework src/NCI.OCPL.Api.Common.Testing/ElasticTools.cs: - Updated using statements: replaced Elasticsearch.Net and Nest with Elastic.Transport and Elastic.Clients.Elasticsearch - Removed Nest.JsonNetSerializer using statement - Changed return types from IElasticClient to ElasticsearchClient - Updated SingleNodeConnectionPool to SingleNodePool (v8 API) - Changed InMemoryConnection to InMemoryRequestInvoker (v8 API) - Updated ConnectionSettings to ElasticsearchClientSettings - Removed sourceSerializer parameter (no longer needed in v8) - Changed ElasticClient instantiation to ElasticsearchClient src/NCI.OCPL.Api.Common.Testing/ElasticsearchInterceptingConnection.cs: - Updated using statements: replaced Elasticsearch.Net with Elastic.Transport - Removed Newtonsoft.Json.Linq, added System.Text.Json - Enabled nullable reference types with #nullable enable - Changed class to implement IRequestInvoker instead of IConnection - Updated ResponseFactory property type from RequestResponseFactory to ResponseFactory - Changed callback signatures from Action<RequestData, ResponseData> to Action<string, ResponseData> - Updated ProcessRequest to work with Endpoint instead of RequestData - Implemented Request<TResponse> and RequestAsync<TResponse> methods with v8 API signatures - Changed GetRequestPost return type from JToken to JsonDocument? - Updated to use System.Text.Json for JSON parsing instead of Newtonsoft.Json - Made properties nullable where appropriate (Stream?, ResponseMimeType?) - Updated PostData.Write call to include disableDirectStreaming parameter src/NCI.OCPL.Api.Common.Testing/ElastcsearchTestingTools.cs: - No functional changes, file was included in staging These changes prepare the testing library for migration from NEST v7 to Elastic.Clients.Elasticsearch v8 and transition from Newtonsoft.Json to System.Text.Json.
…Elasticsearch v8 This commit completes the migration of the testing library from NEST v7 to Elastic.Clients.Elasticsearch v8 and removes all Newtonsoft.Json dependencies. File-by-file changes: src/NCI.OCPL.Api.Common.Testing/NCI.OCPL.Api.Common.Testing.csproj: - Added Microsoft.Extensions.Logging (v8.0.*) package reference This was previously present in v3.1 but was accidentally removed during the initial migration. It's required for NullLogger and NullLoggerOfT classes which provide ILogger, ILogger<T>, LogLevel, and EventId types. src/NCI.OCPL.Api.Common.Testing/ElasticsearchInterceptingConnection.cs: - Removed unused ReadStreamToBytes helper method (no longer needed) - Fixed Request<TResponse> signature to remove CancellationToken parameter to match IRequestInvoker interface requirements - Updated ResponseFactory initialization to use dynamic access to the helper client's Transport.ResponseFactory property - Added ElasticsearchClient helper field to provide access to ResponseFactory - Reformatted ResponseFactory.Create calls for better readability with explicit parameter comments src/NCI.OCPL.Api.Common.Testing/TestingTools.cs: - Updated using statements: removed Newtonsoft.Json.Linq, added System.Text.Json - Renamed GetDataFileAsJObject method to GetDataFileAsJsonDocument - Changed return type from JObject to JsonDocument - Updated implementation to use JsonDocument.Parse instead of JObject.Parse - Updated XML documentation to reflect JsonDocument usage Both NCI.OCPL.Api.Common and NCI.OCPL.Api.Common.Testing projects now build successfully without errors or warnings.
… and System.Text.Json Per-file breakdown: test/NCI.OCPL.Api.Common.Tests/NCI.OCPL.Api.Common.Tests.csproj: - Changed target framework from netcoreapp8.0 to net8.0 - Removed Newtonsoft.Json v13.0.* package reference (no longer needed) test/NCI.OCPL.Api.Common.Tests/Tests/Exceptions/ConfigurationExceptionTest.cs: - Removed redundant 'using NCI.OCPL.Api.Common;' statement (already in correct namespace) test/NCI.OCPL.Api.Common.Tests/Tests/Models/Options/ElasticsearchOptionsTest.cs: - Replaced 'using Newtonsoft.Json' and 'using Newtonsoft.Json.Linq' with 'using System.Text.Json' - Updated Serialize test to use JsonSerializer.Deserialize instead of JToken.Parse for expected value - Updated Serialize test to use Assert.Equivalent with strict: true for object comparison - Updated Deserialize test to use JsonSerializer.Deserialize instead of JsonConvert.DeserializeObject test/NCI.OCPL.Api.Common.Tests/Tests/Models/Options/NSwagOptionsTest.cs: - Replaced 'using Newtonsoft.Json' and 'using Newtonsoft.Json.Linq' with 'using System.Text.Json' - Updated Serialize test to use JsonSerializer.Deserialize instead of JToken.Parse for expected value - Updated Serialize test to use Assert.Equivalent with strict: true for object comparison - Updated Deserialize test to use JsonSerializer.Deserialize instead of JsonConvert.DeserializeObject test/NCI.OCPL.Api.Common.Tests/Tests/NciStartupBase/NciStartupBase.ConfigureServices.cs: - Reorganized using statements into three groups: framework (System.*, Microsoft.*), third-party (Moq, Xunit), solution (Elastic.Clients.Elasticsearch) - Changed IElasticClient references to ElasticsearchClient in ConfigureServices_ElasticsearchBadConfiguration test - Changed IElasticClient references to ElasticsearchClient in ConfigureServices_ElasticsearchGoodConfiguration test - Changed ILogger<ElasticClient> to ILogger<ElasticsearchClient> and Logger<ElasticClient> to Logger<ElasticsearchClient> in ConfigureServices_GetLoggers test
…icsearch v8 and System.Text.Json Per-file breakdown: test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/TestingTools.Test.GetFileAsStream.cs: - Reorganized using statements: grouped System.* together, then blank line, then Xunit test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/TestingTools.Test.GetTestFileAsBytes.cs: - Reorganized using statements: grouped System.* together, then blank line, then Xunit test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/TestingTools.Test.GetStringAsStream.cs: - Reorganized using statements: grouped System.* together, then blank line, then Xunit test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/TestingTools.Test.GetPathToTestFile.cs: - Reorganized using statements: grouped System.* together, then blank line, then Xunit test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/ElastcsearchTestingTools.cs: - Replaced 'using Newtonsoft.Json.Linq' with 'using System.Text.Json' - Changed JToken.Parse to JsonDocument.Parse for expected and actual values - Updated assertion to use Assert.Equivalent with strict: true for JsonDocument comparison test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/ElasticTools._common.cs: - Replaced 'using Nest' with 'using Elastic.Clients.Elasticsearch.Mapping' - TextAttribute now comes from Elastic.Clients.Elasticsearch.Mapping namespace test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/ElasticTools.GetErrorElasticClient.cs: - Replaced 'using Nest' with 'using Elastic.Clients.Elasticsearch' - Changed IElasticClient to ElasticsearchClient - Updated SearchRequest constructor to use index name string instead of Indices object - Updated TermQuery constructor to take field name as parameter - Changed response.IsValid to response.IsValidResponse test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/ElasticTools.GetInMemoryElasticClient.cs: - Replaced 'using Nest' with 'using Elastic.Clients.Elasticsearch' - Removed obsolete [ElasticsearchType] attribute (not needed in v8) - Changed IElasticClient to ElasticsearchClient - Changed response.IsValid to response.IsValidResponse test/NCI.OCPL.Api.Common.Testing.Tests/Tests/Tools/TestingTools.Test.GetDataFileAsJObject.cs: - Replaced 'using Newtonsoft.Json.Linq' with 'using System.Text.Json' - Changed JObject to JsonDocument - Updated method calls from GetDataFileAsJObject to GetDataFileAsJsonDocument - Changed assertion from Assert.Equal with JTokenEqualityComparer to Assert.Equivalent with strict: true
Updates test files to use the new Elastic.Clients.Elasticsearch v8 API:
ElasticTools._common.cs:
- Removed [Text] mapping attribute (no longer exists in v8)
- Removed unused Elastic.Clients.Elasticsearch.Mapping using statement
ElasticTools.GetErrorElasticClient.cs:
- Added Elastic.Clients.Elasticsearch.QueryDsl using statement
- Changed TermQuery from constructor syntax to object initializer syntax
{ Field = "x", Value = "y" } (required in v8)
ElasticTools.GetInMemoryElasticClient.cs:
- Changed Index() to Indices() for SearchTemplateRequestDescriptor
- Changed response.Total to response.Hits.Total.Match(t => t.Value, l => l)
(Total is now Union<TotalHits, long> type in v8)
- Changed response.Documents to response.Hits.Hits with .Source accessor
- Updated assertions to work with Hit<T> wrapper objects
Removed Elasticsearch/ElasticsearchInterceptingConnection tests:
- ElasticsearchInterceptingConnection.Common.cs
- ElasticsearchInterceptingConnection.GetRequestPost.cs
- ElasticsearchInterceptingConnection.ProcessRequest.cs
- ElasticsearchInterceptingConnection.RegisterDefaultHandler.cs
- ElasticsearchInterceptingConnection.RegisterRequestHandlerForType.cs
These tests were removed because they relied heavily on NEST v7 internals
(RequestData, IConnectionConfigurationValues, PostData, Elasticsearch.Net)
that no longer exist in Elastic.Clients.Elasticsearch v8. The
ElasticsearchInterceptingConnection class was already removed from
NCI.OCPL.Api.Common.Testing in a prior migration step.
integration-test-harness.csproj: - Changed TargetFramework from netcoreapp8.0 to net8.0 - Replaced NEST v7.9.* with Elastic.Clients.Elasticsearch v8.19.* Controllers/TestController.cs: - Changed using from Nest to Elastic.Clients.Elasticsearch - Changed IElasticClient to ElasticsearchClient (v8 uses concrete class) - Changed IndexName to Indices for index variable type - Changed IGetResponse to GetResponse (interfaces removed in v8) - Changed IGetRequest to GetRequest (interfaces removed in v8) Controllers/HealthCheckController.cs: - Removed unused Microsoft.Extensions.Options using statement - Removed extra blank line Controllers/ValidRouteController.cs: - Removed unused System.Threading.Tasks using statement - Removed unused NCI.OCPL.Api.Common using statement Program.cs: - Removed unused using statements (System, System.Collections.Generic, System.IO, System.Linq, System.Threading.Tasks, Microsoft.AspNetCore.Hosting, Microsoft.Extensions.Configuration, Microsoft.Extensions.Hosting, Microsoft.Extensions.Logging) Startup.cs: - Removed unused using statements (System, System.Collections.Generic, System.Linq, System.Threading.Tasks, Microsoft.AspNetCore.HttpsPolicy, Microsoft.AspNetCore.Mvc, Microsoft.Extensions.Hosting) Note: CustomSerializationModel.cs still references NEST and Newtonsoft.Json and will be updated separately.
src/NCI.OCPL.Api.Common/Models/ErrorMessage.cs: - Changed from Newtonsoft.Json to System.Text.Json - Changed JsonConvert.SerializeObject to JsonSerializer.Serialize test/integration-test-harness/Program.cs: - Added Microsoft.Extensions.Hosting using for IHost.Run() extension method test/integration-test-harness/Controllers/TestController.cs: - Changed Indices to IndexName (GetRequest expects IndexName in v8) test/integration-test-harness/Models/CustomJsonConverter.cs: - Converted from Newtonsoft.Json to System.Text.Json - Changed ReadJson to Read with Utf8JsonReader signature - Changed WriteJson to Write with Utf8JsonWriter signature - Updated token type checking from ValueType to TokenType - Removed CanWrite property (not used in System.Text.Json) test/integration-test-harness/Models/CustomSerializationModel.cs: - Changed from Nest/Newtonsoft.Json to System.Text.Json.Serialization - Removed [Text] mapping attributes (not needed in v8) - Kept [JsonConverter] attribute for custom serialization .github/copilot-instructions.md: - Added repository coding guidelines for Copilot
The modern Elastic.Clients.Elasticsearch client requires the X-Elastic-Product header to validate the server as a genuine Elasticsearch distribution. Updated both GetInMemoryElasticClient and GetErrorElasticClient to include this header in mock responses.
Replace Assert.Equivalent with JsonNode.DeepEquals to properly compare JSON structures regardless of property order. This provides more reliable JSON comparison in unit tests.
…er than a single property. It doesn't seem possible to use a custom serializer for deserialization only. Going back to JSON requires custom Write.
- Remove array syntax for non-arrray return. - Change custom expectations to reflect clearer test case.
Correct some issues surfaced by the tests.
…constructors and exception handling
- Update coverlet to v6.*, xunit to v2.* Replace async void with async Task to address a deprecation warning for async void.
blairlearn
force-pushed
the
ticket/93-es8
branch
from
2857180 to
d60c0e3
Compare
blairlearn
force-pushed
the
ticket/93-es8
branch
2 times, most recently
from
49fcc40 to
5a75f73
Compare
blairlearn
force-pushed
the
ticket/93-es8
branch
from
5a75f73 to
3365d98
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Many comments to be added.