Skip to content

Latest commit

 

History

History
1154 lines (927 loc) · 56.3 KB

release-notes-elasticsearch-beta1.asciidoc

File metadata and controls

1154 lines (927 loc) · 56.3 KB

{es} version 9.0.0-beta1

Breaking changes

The following changes in {es} 9.0 might affect your applications and prevent them from operating normally. Before upgrading to 9.0, review these changes and take the described steps to mitigate the impact.

Aggregations changes

Remove date histogram boolean support

Details
Elasticsearch no longer allows running Date Histogram aggregations over boolean fields. Instead, use Terms aggregation for boolean fields.

Impact
We expect the impact to be minimal, as this never produced good results, and has been deprecated for years.

Analysis changes

Snowball stemmers have been upgraded

Details
Lucene 10 ships with an upgrade of its Snowball stemmers. For details see apache/lucene#13209. Users using Snowball stemmers that are experiencing changes in search behaviour on existing data are advised to reindex.

Impact
The upgrade should generally provide improved stemming results. Small changes in token analysis can lead to mismatches with previously index data, so existing indices using Snowball stemmers as part of their analysis chain should be reindexed.

The "german2" snowball stemmer is now an alias for the "german" stemmer

Details
Lucene 10 has merged the improved "german2" snowball language stemmer with the "german" stemmer. For Elasticsearch, "german2" is now a deprecated alias for "german". This may results in slightly different tokens being generated for terms with umlaut substitution (like "ue" for "ü" etc…​)

Impact
Replace usages of "german2" with "german" in analysis configuration. Old indices that use the "german" stemmer should be reindexed if possible.

The 'persian' analyzer has stemmer by default

Details
Lucene 10 has added a final stemming step to its PersianAnalyzer that Elasticsearch exposes as 'persian' analyzer. Existing indices will keep the old non-stemming behaviour while new indices will see the updated behaviour with added stemming. Users that wish to maintain the non-stemming behaviour need to define their own analyzer as outlined in {ref}/analysis-lang-analyzer.html#persian-analyzer[Persion analyzer]. Users that wish to use the new stemming behaviour for existing indices will have to reindex their data.

Impact
Indexing with the 'persian' analyzer will produce slightly different tokens. Users should check if this impacts their search results. If they wish to maintain the legacy non-stemming behaviour they can define their own analyzer equivalent as explained in {ref}/analysis-lang-analyzer.html#persian-analyzer[Persian analyzer].

The Korean dictionary for Nori has been updated

Details
Lucene 10 ships with an updated Korean dictionary (mecab-ko-dic-2.1.1). For details see apache/lucene#11452. Users experiencing changes in search behaviour on existing data are advised to reindex.

Impact
The change is small and should generally provide better analysis results. Existing indices for full-text use cases should be reindexed though.

Cluster and node setting changes

Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password will prevent node from starting

Details
For LDAP or AD authentication realms, setting a bind DN (via the xpack.security.authc.realms.ldap..bind_dn or xpack.security.authc.realms.active_directory..bind_dn realm settings) without a bind password is a misconfiguration that may prevent successful authentication to the node. Nodes will fail to start if a bind DN is specified without a password.

Impact
If you have a bind DN configured for an LDAP or AD authentication realm, set a bind password for {ref}/ldap-realm.html#ldap-realm-configuration[LDAP] or {ref}/active-directory-realm.html#ad-realm-configuration[Active Directory]. Configuring a bind DN without a password prevents the misconfigured node from starting.

Deprecated tracing.apm.* settings got removed.

Details
Deprecated tracing.apm. settings got removed, use respective telemetry. / telemetry.tracing.* settings instead.

Impact
9.x nodes will refuse to start if any such setting (including secret settings) is still present.

Limit ByteSizeUnit to 2 decimals

Details
In the past, byte values like 1.25 mb were allowed but deprecated. Now, values with up to two decimal places are allowed, unless the unit is bytes, in which case no decimals are allowed. Values with too many decimal places result in an error.

Impact
Values with more than two decimal places, like 0.123 mb will be rejected as an error, where in the past, they’d be accepted with a deprecation warning.

Minimum shard balancer threshold is now 1.0

Details
Earlier versions of {es} accepted any non-negative value for cluster.routing.allocation.balance.threshold, but values smaller than 1.0 do not make sense and have been ignored since version 8.6.1. From 9.0.0 these nonsensical values are now forbidden.

Impact
Do not set cluster.routing.allocation.balance.threshold to a value less than 1.0.

Remove TLSv1.1 from default protocols

Details
TLSv1.1 is no longer enabled by default. Prior to version 9.0, Elasticsearch would attempt to enable TLSv1.1 if the JDK supported it. In most cases, including all cases where Elasticsearch 8 was running with the bundled JDK, the JDK would not support TLSv1.1, so that protocol would not be available in Elasticsearch. However, if Elasticsearch was running on an old JDK or a JDK that have been reconfigured to support TLSv1.1, then the protocol would automatically be available within Elasticsearch. As of Elasticsearch 9.0, this is no longer true. If you wish to enable TLSv1.1 then you must enable it within the JDK and also enable it within Elasticsearch by using the ssl.supported_protocols setting.

Impact
Most users will not be impacted. If your Elastisearch 8 cluster was using a custom JDK and you relied on TLSv1.1, then you will need to explicitly enable TLSv1.1 within Elasticsearch (as well as enabling it within your JDK)

Remove client.type setting

Details
The node setting client.type has been ignored since the node client was removed in 8.0. The setting is now removed.

Impact
Remove the client.type setting from elasticsearch.yml

Remove cluster.routing.allocation.disk.watermark.enable_for_single_data_node setting

Details
Prior to 7.8, whenever a cluster had only a single data node, the watermarks would not be respected. In order to change this in 7.8+ in a backwards compatible way, we introduced the cluster.routing.allocation.disk.watermark.enable_for_single_data_node node setting. The setting was deprecated in 7.14 and was made to accept only true in 8.0

Impact
No known end user impact

Remove deprecated xpack.searchable.snapshot.allocate_on_rolling_restart setting

Details
The xpack.searchable.snapshot.allocate_on_rolling_restart setting was created as an escape-hatch just in case relying on the cluster.routing.allocation.enable=primaries setting for allocating searchable snapshots during rolling restarts had some unintended side-effects. It has been deprecated since 8.2.0.

Impact
Remove xpack.searchable.snapshot.allocate_on_rolling_restart from your settings if present.

Remove unsupported legacy value for discovery.type

Details
Earlier versions of {es} had a discovery.type setting which permitted values that referred to legacy discovery types. From v9.0.0 onwards, the only supported values for this setting are multi-node (the default) and single-node.

Impact
Remove any value for discovery.type from your elasticsearch.yml configuration file.

Index setting changes

Remove the ability to read frozen indices

Details
The ability to read frozen indices has been removed. (Frozen indices are no longer useful due to improvements in heap memory usage. The ability to freeze indices was removed in 8.0.)

Impact
Users must unfreeze any frozen indices before upgrading.

Ingest changes

Remove ecs option on user_agent processor

Details
The user_agent ingest processor no longer accepts the ecs option. (It was previously deprecated and ignored.)

Impact
Users should stop using the ecs option when creating instances of the user_agent ingest processor. The option will be removed from existing processors stored in the cluster state on upgrade.

Remove ignored fallback option on GeoIP processor

Details
The option fallback_to_default_databases on the geoip ingest processor has been removed. (It was deprecated and ignored since 8.0.0.)

Impact
Customers should stop remove the noop fallback_to_default_databases option on any geoip ingest processors.

Logging changes

Deprecation logging value change for "data_stream.dataset" and "event.dataset"

Details
This change modifies the "data_stream.dataset" and "event.dataset" value for deprecation logging to use the value elasticsearch.deprecation instead of deprecation.elasticsearch. This is now consistent with other values where the name of the service is the first part of the key.

Impact
If you are directly consuming deprecation logs for "data_stream.dataset" and "event.dataset" and filtering on this value, you will need to update your filters to use elasticsearch.deprecation instead of deprecation.elasticsearch.

Logs changes

Conditionally enable logsdb by default

Details
Logsdb will be enabled by default for data streams matching with logs-- pattern. If upgrading from 8.x to 9.x and data streams matching with log-- do exist, then Logsdb will not be enabled by default.

Impact
Logsdb reduce storage footprint in Elasticsearch for logs, but there are side effects to be taken into account that are described in the {ref}/logs-data-stream.html[Logsdb docs].

Mapping changes

Remove support for type, fields, copy_to and boost in metadata field definition

Details
The type, fields, copy_to and boost parameters are no longer supported in metadata field definition starting with version 9.

Impact
Users providing type, fields, copy_to or boost as part of metadata field definition should remove them from their mappings.

Turn _source meta fieldmapper’s mode attribute into a no-op

Details
The mode mapping attribute of _source metadata field mapper has been turned into a no-op. Instead the index.mapping.source.mode index setting should be used to configure source mode.

Impact
Configuring the mode attribute for the _source meta field mapper will have no effect on indices created with Elasticsearch 9.0.0 or later. Note that _source.mode configured on indices before upgrading to 9.0.0 or later will remain efficive after upgrading.

Packaging changes

Disable machine learning on macOS x86_64

Details
The machine learning plugin is permanently disabled on macOS x86_64. For the last three years Apple has been selling hardware based on the arm64 architecture, and support will increasingly focus on this architecture in the future. Changes to upstream dependencies of Elastic’s machine learning functionality have made it unviable for Elastic to continue to build machine learning on macOS x86_64.

Impact
To continue to use machine learning functionality on macOS please switch to an arm64 machine (Apple silicon). Alternatively, it will still be possible to run Elasticsearch with machine learning enabled in a Docker container on macOS x86_64.

REST API changes

Apply more strict parsing of actions in bulk API

Details
Previously, the following classes of malformed input were deprecated but not rejected in the action lines of the a bulk request: missing closing brace; additional keys after the action (which were ignored); additional data after the closing brace (which was ignored). They will now be considered errors and rejected.

Impact
Users must provide well-formed input when using the bulk API. (They can request REST API compatibility with v8 to get the previous behaviour back as an interim measure.)

Change most Elasticsearch timeouts to 429 response instead of 5xx

Details
When a timeout occurs in most REST requests, whether via a per-request timeout, or a system default, the request would return a 5xx response code. The response code from those APIs when a timeout occurs is now 429.

Impact
Adjust any code relying on retrying on 5xx responses for timeouts to look for a 429 response code and inspect the response to determine whether a timeout occured.

Error JSON structure has changed when detailed errors are disabled

Details
This change modifies the JSON format of error messages returned to REST clients when detailed messages are turned off. Previously, JSON returned when an exception occurred, and http.detailed_errors.enabled: false was set, just consisted of a single "error" text field with some basic information. Setting http.detailed_errors.enabled: true (the default) changed this field to an object with more detailed information. With this change, non-detailed errors now have the same structure as detailed errors. "error" will now always be an object with, at a minimum, a "type" and "reason" field. Additional fields are included when detailed errors are enabled. To use the previous structure for non-detailed errors, use the v8 REST API.

Impact
If you have set http.detailed_errors.enabled: false (the default is true) the structure of JSON when any exceptions occur now matches the structure when detailed errors are enabled. To use the previous structure for non-detailed errors, use the v8 REST API.

Remove any references to org.elasticsearch.core.RestApiVersion#V_7

Details
This PR removes all references to V_7 in the Rest API. V7 features marked for deprecation have been removed.

Impact
This change is breaking for any external plugins/clients that rely on the V_7 enum or deprecated version 7 functionality

Remove cluster state from /_cluster/reroute response

Details
The POST /_cluster/reroute API no longer returns the cluster state in its response. The ?metric query parameter to this API now has no effect and its use will be forbidden in a future version.

Impact
Cease usage of the ?metric query parameter when calling the POST /_cluster/reroute API.

Remove deprecated local attribute from alias APIs

Details
The following APIs no longer accept the ?local query parameter: GET /_alias, GET /_aliases, GET /_alias/{name}, HEAD /_alias/{name}, GET /{index}/_alias, HEAD /{index}/_alias, GET /{index}/_alias/{name}, HEAD /{index}/_alias/{name}, GET /_cat/aliases, and GET /_cat/aliases/{alias}. This parameter has been deprecated and ignored since version 8.12.

Impact
Cease usage of the ?local query parameter when calling the listed APIs.

Remove legacy params from range query

Details
The deprecated range query parameters to, from, include_lower, and include_upper are no longer supported.

Impact
Users should use lt, lte, gt, and gte query parameters instead.

Remove old _knn_search tech preview API in v9

Details
The original, tech-preview api for vector search, _knn_search, has been removed in v9. For all vector search operations, you should utilize the _search endpoint.

Impact
The _knn_search API is now inaccessible without providing a compatible-with flag for v8.

Remove support for deprecated force_source highlighting parameter

Details
The deprecated highlighting force_source parameter is no longer supported.

Impact
Users should remove usages of the force_source parameter from their search requests.

Remove unfreeze REST endpoint

Details
The /{index}/_unfreeze REST endpoint is no longer supported. This API was deprecated, and the corresponding /{index}/_freeze endpoint was removed in 8.0.

Impact
None, since it is not possible to have a frozen index in a version which is readable by Elasticsearch 9.0

Removing support for types field in watcher search

Details
Previously, setting the input.search.request.types field in the payload when creating a watcher to an empty array was allowed, although it resulted in a deprecation warning and had no effect (and any value other than an empty array would result in an error). Now, support for this field is entirely removed, and the empty array will also result in an error.

Impact
Users should stop setting this field (which did not have any effect anyway).

Restrict Connector APIs to manage/monitor_connector privileges

Details
Connector APIs now enforce the manage_connector and monitor_connector privileges (introduced in 8.15), replacing the previous reliance on index-level permissions for .elastic-connectors and .elastic-connectors-sync-jobs in API calls.

Impact
Connector APIs now require manage_connector and monitor_connector privileges

Set allow_partial_search_results=true by default

Details
Before this change, in case of shard failures, EQL queries always returned an error. With this change, they will keep running and will return partial results.

Impact
EQL queries that would previously fail due to shard failures, will now succeed and return partial results. The previous defaults can be restored by setting xpack.eql.default_allow_partial_results cluster setting to false or setting with allow_partial_search_results to false in the query request.

Search changes

Adjust random_score default field to _seq_no field

Details
When providing a 'seed' parameter to a 'random_score' function in the 'function_score' query but NOT providing a 'field', the default 'field' is switched from '_id' to '_seq_no'.

Impact
The random scoring and ordering may change when providing a 'seed' and not providing a 'field' to a 'random_score' function.

Change semantic text to act like a normal text field

Details
The previous semantic_text format used a complex subfield structure in _source to store the embeddings. This complicated interactions/integrations with semantic_text fields and _source in general. This new semantic_text format treats it as a normal text field, where the field’s value in _source is the value assigned by the user.

Impact
Users who parsed the subfield structure of the previous semantic_text format in _source will need to update their parsing logic. The new format does not directly expose the chunks and embeddings generated from the input text. The new format will be applied to all new indices, any existing indices will continue to use the previous format.

Transform changes

Remove data_frame_transforms roles

Details
data_frame_transforms_admin and data_frame_transforms_user were deprecated in Elasticsearch 7 and are being removed in Elasticsearch 9. data_frame_transforms_admin is now transform_admin. data_frame_transforms_user is now transform_user. Users must call the _update API to replace the permissions on the Transform before the Transform can be started.

Impact
Transforms created with either the data_frame_transforms_admin or the data_frame_transforms_user role will fail to start. The Transform will remain in a stopped state, and its health will be red while displaying permission failures.

The following pull requests relate to breaking changes:

Aggregations

  • Remove date histogram boolean support {es-pull}118484[#118484]

Allocation

  • Increase minimum threshold in shard balancer {es-pull}115831[#115831]

  • Remove cluster.routing.allocation.disk.watermark.enable_for_single_data_node setting {es-pull}114207[#114207]

  • Remove cluster state from /_cluster/reroute response {es-pull}114231[#114231] (issue: {es-issue}88978[#88978])

Analysis

  • Snowball stemmers have been upgraded {es-pull}114146[#114146]

  • The 'german2' stemmer is now an alias for the 'german' snowball stemmer {es-pull}113614[#113614]

  • The 'persian' analyzer has stemmer by default {es-pull}113482[#113482] (issue: {es-issue}113050[#113050])

  • The Korean dictionary for Nori has been updated {es-pull}114124[#114124]

Authentication

  • Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password will prevent node from starting {es-pull}118366[#118366]

Cluster Coordination

  • Remove unsupported legacy value for discovery.type {es-pull}112903[#112903]

EQL

  • Set allow_partial_search_results=true by default {es-pull}120267[#120267]

Extract&Transform

  • Restrict Connector APIs to manage/monitor_connector privileges {es-pull}119863[#119863]

Highlighting

  • Remove support for deprecated force_source highlighting parameter {es-pull}116943[#116943]

Indices APIs

  • Apply more strict parsing of actions in bulk API {es-pull}115923[#115923]

  • Remove deprecated local attribute from alias APIs {es-pull}115393[#115393]

  • Remove the ability to read frozen indices {es-pull}120108[#120108]

  • Remove unfreeze REST endpoint {es-pull}119227[#119227]

Infra/Core

  • Change Elasticsearch timeouts to 429 response instead of 5xx {es-pull}116026[#116026]

  • Limit ByteSizeUnit to 2 decimals {es-pull}120142[#120142]

  • Remove client.type setting {es-pull}118192[#118192] (issue: {es-issue}104574[#104574])

  • Remove any references to org.elasticsearch.core.RestApiVersion#V_7 {es-pull}118103[#118103]

Infra/Logging

  • Change deprecation.elasticsearch keyword to elasticsearch.deprecation {es-pull}117933[#117933] (issue: {es-issue}83251[#83251])

Infra/Metrics

  • Deprecated tracing.apm.* settings got removed. {es-pull}119926[#119926]

Infra/REST API

  • Output a consistent format when generating error json {es-pull}90529[#90529] (issue: {es-issue}89387[#89387])

Ingest Node

  • Remove ecs option on user_agent processor {es-pull}116077[#116077]

  • Remove ignored fallback option on GeoIP processor {es-pull}116112[#116112]

Logs

  • Conditionally enable logsdb by default for data streams matching with logs-- pattern. {es-pull}121049[#121049] (issue: {es-issue}106489[#106489])

Machine Learning

  • Disable machine learning on macOS x86_64 {es-pull}104125[#104125]

Mapping

  • Remove support for type, fields, copy_to and boost in metadata field definition {es-pull}118825[#118825]

  • Turn _source meta fieldmapper’s mode attribute into a no-op {es-pull}119072[#119072] (issue: {es-issue}118596[#118596])

Search

  • Adjust random_score default field to _seq_no field {es-pull}118671[#118671]

  • Change Semantic Text To Act Like A Normal Text Field {es-pull}120813[#120813]

  • Remove legacy params from range query {es-pull}116970[#116970]

Snapshot/Restore

  • Remove deprecated xpack.searchable.snapshot.allocate_on_rolling_restart setting {es-pull}114202[#114202]

TLS

  • Remove TLSv1.1 from default protocols {es-pull}121731[#121731]

Transform

  • Remove data_frame_transforms roles {es-pull}117519[#117519]

Vector Search

  • Remove old _knn_search tech preview API in v9 {es-pull}118104[#118104]

Watcher

  • Removing support for types field in watcher search {es-pull}120748[#120748]

Deprecations

The following functionality has been deprecated in {es} 9.0 and will be removed in a future version. While this won’t have an immediate impact on your applications, we strongly encourage you to take the described steps to update your code after upgrading to 9.0.

To find out if you are using any deprecated functionality, enable {ref}/logging.html#deprecation-logging[deprecation logging].

Authorization deprecations

Deprecate certificate based remote cluster security model

Details
-| {ref}/remote-clusters-cert.html[Certificate-based remote cluster security model] is deprecated and will be removed in a future major version. Users are encouraged to {ref}/remote-clusters-migrate.html[migrate remote clusters from certificate to API key authentication]. The {ref}/remote-clusters-api-key.html[API key-based security model] is preferred way to configure remote clusters, as it allows to follow security best practices when setting up remote cluster connections and defining fine-grained access control.

Impact
-| If you have configured remote clusters with certificate-based security model, you should {ref}/remote-clusters-migrate.html[migrate remote clusters from certificate to API key authentication]. Configuring a remote cluster using {ref}/remote-clusters-cert.html[certificate authentication], generates a warning in the deprecation logs.

ES|QL deprecations

Drop support for brackets from METADATA syntax

Details
Please describe the details of this change for the release notes. You can use asciidoc.

Impact
Please describe the impact of this change to users

REST API deprecations

Document _type deprecated on simulate pipeline API

Details
Passing a document with a _type property is deprecated in the /_ingest/pipeline/{id}/_simulate and /_ingest/pipeline/_simulate APIs.

Impact
Users should already have stopped using mapping types, which were deprecated in {es} 7. This deprecation warning will fire if they specify mapping types on documents pass to the simulate pipeline API.

Deprecate elser service

Details
The elser service of the inference API will be removed in an upcoming release. Please use the elasticsearch service instead.

Impact
In the current version there is no impact. In a future version, users of the elser service will no longer be able to use it, and will be required to use the elasticsearch service to access elser through the inference API.

Rollup deprecations

Emit deprecation warning when executing one of the rollup APIs

Details
Rollup is already deprecated since 8.11.0 via documentation and since 8.15.0 it is no longer possible to create new rollup jobs in clusters without rollup usage. This change updates the rollup APIs to emit a deprecation warning.

Impact
Returning a deprecation warning when using one of the rollup APIs.

Transform deprecations

Removing index alias creation for deprecated transforms notification index

Details
As part of the migration from 7.x to 8.x, the .data-frame-notifications-1 index was deprecated and replaced with the .transform-notifications-000002 index. The index is no longer created by default, all writes are directed to the new index, and any clusters with the deprecated index will have an alias created to ensure that reads are still retrieving data that was written to the index before the migration to 8.x. This change removes the alias from the deprecated index in 9.x. Any clusters with the alias present will retain it, but it will not be created on new clusters.

Impact
No known end user impact.

The following pull requests relate to deprecations:

ES|QL

  • Drop support for brackets from METADATA syntax {es-pull}119846[#119846] (issue: {es-issue}115401[#115401])

Ingest Node

  • Fix _type deprecation on simulate pipeline API {es-pull}116259[#116259]

Machine Learning

  • Removing index alias creation for deprecated transforms notification index {es-pull}117583[#117583]

  • [Inference API] Deprecate elser service {es-pull}113216[#113216]

Rollup

  • Emit deprecation warning when executing one of the rollup APIs {es-pull}113131[#113131]

Security

  • Deprecate certificate based remote cluster security model {es-pull}120806[#120806]

Bug fixes

Aggregations

  • Handle with illegalArgumentExceptions negative values in HDR percentile aggregations {es-pull}116174[#116174] (issue: {es-issue}115777[#115777])

Analysis

  • Analyze API to return 400 for wrong custom analyzer {es-pull}121568[#121568] (issue: {es-issue}121443[#121443])

CAT APIs

  • Fix cat_component_templates documentation {es-pull}120487[#120487]

CRUD

  • Preserve thread context when waiting for segment generation in RTG {es-pull}114623[#114623]

  • Preserve thread context when waiting for segment generation in RTG {es-pull}117148[#117148]

Data streams

  • Avoid updating settings version in MetadataMigrateToDataStreamService when settings have not changed {es-pull}118704[#118704]

  • Block-writes cannot be added after read-only {es-pull}119007[#119007] (issue: {es-issue}119002[#119002])

  • Ensure removal of index blocks does not leave key with null value {es-pull}122246[#122246]

  • Match dot prefix of migrated DS backing index with the source index {es-pull}120042[#120042]

  • Refresh source index before reindexing data stream index {es-pull}120752[#120752] (issue: {es-issue}120314[#120314])

  • ReindexDataStreamIndex bug in assertion caused by reference equality {es-pull}121325[#121325]

Downsampling

  • Copy metrics and default_metric properties when downsampling aggregate_metric_double {es-pull}121727[#121727] (issues: {es-issue}119696[#119696], {es-issue}96076[#96076])

EQL

  • Fix JOIN command validation (not supported) {es-pull}122011[#122011]

ES|QL

  • Allow the data type of null in filters {es-pull}118324[#118324] (issue: {es-issue}116351[#116351])

  • Correct line and column numbers of missing named parameters {es-pull}120852[#120852]

  • Drop null columns in text formats {es-pull}117643[#117643] (issue: {es-issue}116848[#116848])

  • Fix ENRICH validation for use of wildcards {es-pull}121911[#121911]

  • Fix ROUND() with unsigned longs throwing in some edge cases {es-pull}119536[#119536]

  • Fix TDigestState.read CB leaks {es-pull}114303[#114303] (issue: {es-issue}114194[#114194])

  • Fix TopN row size estimate {es-pull}119476[#119476] (issue: {es-issue}106956[#106956])

  • Fix AbstractShapeGeometryFieldMapperTests {es-pull}119265[#119265] (issue: {es-issue}119201[#119201])

  • Fix a bug in TOP {es-pull}121552[#121552]

  • Fix async stop sometimes not properly collecting result {es-pull}121843[#121843] (issue: {es-issue}121249[#121249])

  • Fix attribute set equals {es-pull}118823[#118823]

  • Fix double lookup failure on ESQL {es-pull}115616[#115616] (issue: {es-issue}111398[#111398])

  • Fix queries with document level security on lookup indexes {es-pull}120617[#120617] (issue: {es-issue}120509[#120509])

  • Fix writing for LOOKUP status {es-pull}119296[#119296] (issue: {es-issue}119086[#119086])

  • Limit memory usage of fold {es-pull}118602[#118602]

  • Limit size of query {es-pull}117898[#117898]

  • Remove redundant sorts from execution plan {es-pull}121156[#121156]

  • Revert unwanted ES|QL lexer changes from PR #120354 {es-pull}120538[#120538]

Health

  • Do not recommend increasing max_shards_per_node {es-pull}120458[#120458]

Indices APIs

  • Add ?master_timeout to POST /_ilm/migrate_to_data_tiers {es-pull}120883[#120883]

  • Fix broken yaml test 30_create_from {es-pull}120662[#120662]

  • Include hidden indices in DeprecationInfoAction {es-pull}118035[#118035] (issue: {es-issue}118020[#118020])

Inference

  • [Inference API] Put back legacy EIS URL setting {es-pull}121207[#121207]

Infra/Core

  • Epoch Millis Rounding Down and Not Up 2 {es-pull}118353[#118353]

  • Wrap jackson exception on malformed json string {es-pull}114445[#114445] (issue: {es-issue}114142[#114142])

Infra/Logging

  • Move SlowLogFieldProvider instantiation to node construction {es-pull}117949[#117949]

Infra/Metrics

  • Make randomInstantBetween always return value in range [minInstant, maxInstant] {es-pull}114177[#114177]

Infra/Plugins

  • Remove unnecessary entitlement {es-pull}120959[#120959]

  • Restrict agent entitlements to the system classloader unnamed module {es-pull}120546[#120546]

Infra/REST API

  • Fixed a NullPointerException in _capabilities API when the path parameter is null. {es-pull}113413[#113413] (issue: {es-issue}113413[#113413])

Infra/Scripting

  • Register mustache size limit setting {es-pull}119291[#119291]

Infra/Settings

  • Don’t allow secure settings in YML config (109115) {es-pull}115779[#115779] (issue: {es-issue}109115[#109115])

Ingest Node

  • Add warning headers for ingest pipelines containing special characters {es-pull}114837[#114837] (issue: {es-issue}104411[#104411])

  • Fix geoip databases index access after system feature migration {es-pull}121196[#121196]

Logs

  • Always check if index mode is logsdb {es-pull}116922[#116922]

Machine Learning

  • Change format for Unified Chat {es-pull}121396[#121396]

  • Fix get all inference endponts not returning multiple endpoints sharing model deployment {es-pull}121821[#121821]

  • Fixing bedrock event executor terminated cache issue {es-pull}118177[#118177] (issue: {es-issue}117916[#117916])

  • Fixing bug setting index when parsing Google Vertex AI results {es-pull}117287[#117287]

  • Updating Inference Update API documentation to have the correct PUT method {es-pull}121048[#121048]

  • Wait for up to 2 seconds for yellow status before starting search {es-pull}115938[#115938] (issues: {es-issue}107777[#107777], {es-issue}105955[#105955], {es-issue}107815[#107815], {es-issue}112191[#112191])

  • [Inference API] Fix unique ID message for inference ID matches trained model ID {es-pull}119543[#119543] (issue: {es-issue}111312[#111312])

Mapping

  • Enable New Semantic Text Format Only On Newly Created Indices {es-pull}121556[#121556]

  • Fix propagation of dynamic mapping parameter when applying copy_to {es-pull}121109[#121109] (issue: {es-issue}113049[#113049])

  • Fix realtime get of nested fields with synthetic source {es-pull}119575[#119575] (issue: {es-issue}119553[#119553])

  • Merge field mappers when updating mappings with [subobjects:false] {es-pull}120370[#120370] (issue: {es-issue}120216[#120216])

  • Tweak copy_to handling in synthetic _source to account for nested objects {es-pull}120974[#120974] (issue: {es-issue}120831[#120831])

Network

  • Remove ChunkedToXContentBuilder {es-pull}119310[#119310] (issue: {es-issue}118647[#118647])

Search

  • Catch and handle disconnect exceptions in search {es-pull}115836[#115836]

  • Fix leak in DfsQueryPhase and introduce search disconnect stress test {es-pull}116060[#116060] (issue: {es-issue}115056[#115056])

  • Fix/QueryBuilderBWCIT_muted_test {es-pull}117831[#117831]

  • In this pr, a 400 error is returned when _source / _seq_no / _feature / _nested_path / _field_names is requested, rather a 5xx {es-pull}117229[#117229]

  • Inconsistency in the _analyzer api when the index is not included {es-pull}115930[#115930]

  • Remove duplicate code in ESIntegTestCase {es-pull}120799[#120799]

  • SearchStatesIt failures reported by CI {es-pull}117618[#117618] (issues: {es-issue}116617[#116617], {es-issue}116618[#116618])

  • Skip fetching _inference_fields field in legacy semantic_text format {es-pull}121720[#121720]

  • Test/107515 restore template with match only text mapper it fail {es-pull}120392[#120392] (issue: {es-issue}107515[#107515])

  • Updated Date Range to Follow Documentation When Assuming Missing Values {es-pull}112258[#112258] (issue: {es-issue}111484[#111484])

  • CrossClusterIT testCancel failure {es-pull}117750[#117750] (issue: {es-issue}108061[#108061])

  • SearchServiceTests.testParseSourceValidation failure {es-pull}117963[#117963]

Snapshot/Restore

  • Add undeclared Azure settings, modify test to exercise them {es-pull}118634[#118634]

  • Fork post-snapshot-delete cleanup off master thread {es-pull}122047[#122047]

  • Retry throttled snapshot deletions {es-pull}113237[#113237]

  • Use the system index descriptor in the snapshot blob cache cleanup task {es-pull}120937[#120937] (issue: {es-issue}120518[#120518])

Store

  • Do not capture ClusterChangedEvent in IndicesStore call to #onClusterStateShardsClosed {es-pull}120193[#120193]

Vector Search

  • Apply default k for knn query eagerly {es-pull}118774[#118774]

  • Fix bbq_hnsw merge file cleanup on random IO exceptions {es-pull}119691[#119691] (issue: {es-issue}119392[#119392])

Watcher

  • Watcher history index has too many indexed fields - {es-pull}117701[#117701] (issue: {es-issue}71479[#71479])

Enhancements

Allocation

  • Add a not-master state for desired balance {es-pull}116904[#116904]

  • Only publish desired balance gauges on master {es-pull}115383[#115383]

  • Reset relocation/allocation failure counter on node join/shutdown {es-pull}119968[#119968]

Authentication

  • Allow SSHA-256 for API key credential hash {es-pull}120997[#120997]

Authorization

  • Allow kibana_system user to manage .reindexed-v8-internal.alerts indices {es-pull}118959[#118959]

  • Grant necessary Kibana application privileges to reporting_user role {es-pull}118058[#118058]

  • Make reserved built-in roles queryable {es-pull}117581[#117581]

  • [Security Solution] Add create_index to kibana_system role for index/DS .logs-endpoint.action.responses-* {es-pull}115241[#115241]

  • [Security Solution] allows kibana_system user to manage .reindexed-v8-* Security Solution indices {es-pull}119054[#119054]

CCS

  • Resolve/cluster allows querying for cluster info only (no index expression required) {es-pull}119898[#119898]

CRUD

  • Remove INDEX_REFRESH_BLOCK after index becomes searchable {es-pull}120807[#120807]

  • Suppress merge-on-recovery for older indices {es-pull}113462[#113462]

Cluster Coordination

  • Include clusterApplyListener in long cluster apply warnings {es-pull}120087[#120087]

Data streams

  • Add action to create index from a source index {es-pull}118890[#118890]

  • Add index and reindex request settings to speed up reindex {es-pull}119780[#119780]

  • Add rest endpoint for create_from_source_index {es-pull}119250[#119250]

  • Add sanity check to ReindexDatastreamIndexAction {es-pull}120231[#120231]

  • Adding a migration reindex cancel API {es-pull}118291[#118291]

  • Adding get migration reindex status {es-pull}118267[#118267]

  • Consistent mapping for OTel log and event bodies {es-pull}120547[#120547]

  • Filter deprecated settings when making dest index {es-pull}120163[#120163]

  • Ignore closed indices for reindex {es-pull}120244[#120244]

  • Improve how reindex data stream index action handles api blocks {es-pull}120084[#120084]

  • Initial work on ReindexDatastreamIndexAction {es-pull}116996[#116996]

  • Make requests_per_second configurable to throttle reindexing {es-pull}120207[#120207]

  • Optimized index sorting for OTel logs {es-pull}119504[#119504]

  • Report Deprecated Indices That Are Flagged To Ignore Migration Reindex As A Warning {es-pull}120629[#120629]

  • Update data stream deprecations warnings to new format and filter searchable snapshots from response {es-pull}118562[#118562]

Distributed

  • Make various alias retrieval APIs wait for cluster to unblock {es-pull}117230[#117230]

  • Metrics for incremental bulk splits {es-pull}116765[#116765]

  • Use Azure blob batch API to delete blobs in batches {es-pull}114566[#114566]

EQL

  • Add support for partial shard results {es-pull}116388[#116388]

  • Optional named arguments for function in map {es-pull}118619[#118619]

ES|QL

  • Add ES|QL cross-cluster query telemetry collection {es-pull}119474[#119474]

  • Add a LicenseAware interface for licensed Nodes {es-pull}118931[#118931] (issue: {es-issue}117405[#117405])

  • Add a PostAnalysisAware, distribute verification {es-pull}119798[#119798]

  • Add a standard deviation aggregating function: STD_DEV {es-pull}116531[#116531]

  • Add cluster level reduction {es-pull}117731[#117731]

  • Add nulls support to Categorize {es-pull}117655[#117655]

  • Async search responses have CCS metadata while searches are running {es-pull}117265[#117265]

  • Check for early termination in Driver {es-pull}118188[#118188]

  • Do not serialize EsIndex in plan {es-pull}119580[#119580]

  • ESQL - Remove restrictions for disjunctions in full text functions {es-pull}118544[#118544]

  • ESQL - enabling scoring with METADATA _score {es-pull}113120[#113120]

  • ESQL Add esql hash function {es-pull}117989[#117989]

  • ESQL Support IN operator for Date nanos {es-pull}119772[#119772] (issue: {es-issue}118578[#118578])

  • ESQL: CATEGORIZE as a BlockHash {es-pull}114317[#114317]

  • ESQL: Enterprise license enforcement for CCS {es-pull}118102[#118102]

  • ES|QL: Partial result on demand for async queries {es-pull}118122[#118122]

  • Enable KQL function as a tech preview {es-pull}119730[#119730]

  • Enable LOOKUP JOIN in non-snapshot builds {es-pull}121193[#121193] (issue: {es-issue}121185[#121185])

  • Enable node-level reduction by default {es-pull}119621[#119621]

  • Enable physical plan verification {es-pull}118114[#118114]

  • Ensure cluster string could be quoted {es-pull}120355[#120355]

  • Esql - Support date nanos in date extract function {es-pull}120727[#120727] (issue: {es-issue}110000[#110000])

  • Esql - support date nanos in date format function {es-pull}120143[#120143] (issue: {es-issue}109994[#109994])

  • Esql Support date nanos on date diff function {es-pull}120645[#120645] (issue: {es-issue}109999[#109999])

  • Esql bucket function for date nanos {es-pull}118474[#118474] (issue: {es-issue}118031[#118031])

  • Esql compare nanos and millis {es-pull}118027[#118027] (issue: {es-issue}116281[#116281])

  • Esql implicit casting for date nanos {es-pull}118697[#118697] (issue: {es-issue}118476[#118476])

  • Extend TranslationAware to all pushable expressions {es-pull}120192[#120192]

  • Hash functions {es-pull}118938[#118938]

  • Implement a MetricsAware interface {es-pull}121074[#121074]

  • LOOKUP JOIN using field-caps for field mapping {es-pull}117246[#117246]

  • Lookup join on multiple join fields not yet supported {es-pull}118858[#118858]

  • Move scoring in ES|QL out of snapshot {es-pull}120354[#120354]

  • Optimize ST_EXTENT_AGG for geo_shape and cartesian_shape {es-pull}119889[#119889]

  • Push down filter passed lookup join {es-pull}118410[#118410]

  • Resume Driver on cancelled or early finished {es-pull}120020[#120020]

  • Rewrite TO_UPPER/TO_LOWER comparisons {es-pull}118870[#118870] (issue: {es-issue}118304[#118304])

  • ST_EXTENT_AGG optimize envelope extraction from doc-values for cartesian_shape {es-pull}118802[#118802]

  • Smarter field caps with subscribable listener {es-pull}116755[#116755]

  • Support partial sort fields in TopN pushdown {es-pull}116043[#116043] (issue: {es-issue}114515[#114515])

  • Support some stats on aggregate_metric_double {es-pull}120343[#120343] (issue: {es-issue}110649[#110649])

  • Take named parameters for identifier and pattern out of snapshot {es-pull}121850[#121850]

  • Term query for ES|QL {es-pull}117359[#117359]

  • Update grammar to rely on indexPattern instead of identifier in join target {es-pull}120494[#120494]

  • _score should not be a reserved attribute in ES|QL {es-pull}118435[#118435] (issue: {es-issue}118460[#118460])

Engine

  • Defer unpromotable shard refreshes until index refresh blocks are cleared {es-pull}120642[#120642]

  • POC mark read-only {es-pull}119743[#119743]

Experiences

  • Integrate IBM watsonx to Inference API for re-ranking task {es-pull}117176[#117176]

Geo

  • Optimize indexing points with index and doc values set to true {es-pull}120271[#120271]

Health

  • Increase replica_unassigned_buffer_time default from 3s to 5s {es-pull}112834[#112834]

ILM+SLM

  • Add a replicate_for option to the ILM searchable_snapshot action {es-pull}119003[#119003]

Indices APIs

  • Add remove_index_block arg to _create_from api {es-pull}120548[#120548]

  • Remove index blocks by default in create_from {es-pull}120643[#120643]

  • Run TransportGetComponentTemplateAction on local node {es-pull}116868[#116868]

  • Run TransportGetComposableIndexTemplate on local node {es-pull}119830[#119830]

  • Run TransportGetIndexTemplateAction on local node {es-pull}119837[#119837]

  • introduce new categories for deprecated resources in deprecation API {es-pull}120505[#120505]

Inference

  • Add version prefix to Inference Service API path {es-pull}117095[#117095]

  • Remove Elastic Inference Service feature flag and deprecated setting {es-pull}120842[#120842]

  • Update sparse text embeddings API route for Inference Service {es-pull}118025[#118025]

  • [Elastic Inference Service] Add ElasticInferenceService Unified ChatCompletions Integration {es-pull}118871[#118871]

Infra/CLI

  • Strengthen encryption for elasticsearch-keystore tool to AES 256 {es-pull}119749[#119749]

Infra/Circuit Breakers

  • Add link to Circuit Breaker "Data too large" exception message {es-pull}113561[#113561]

Infra/Core

  • Add support for specifying reindexing script for system index migration {es-pull}119001[#119001]

  • Change default Docker image to be based on UBI minimal instead of Ubuntu {es-pull}116739[#116739]

Infra/Metrics

  • Add ensureGreen test method for use with adminClient {es-pull}113425[#113425]

Infra/REST API

  • A new query parameter ?include_source_on_error was added for create / index, update and bulk REST APIs to control if to include the document source in the error response in case of parsing errors. The default value is true. {es-pull}120725[#120725]

Infra/Scripting

  • Add a mustache.max_output_size_bytes setting to limit the length of results from mustache scripts {es-pull}114002[#114002]

Infra/Settings

  • Introduce IndexSettingDeprecatedInV8AndRemovedInV9 Setting property {es-pull}120334[#120334]

  • Run TransportClusterGetSettingsAction on local node {es-pull}119831[#119831]

Ingest Node

  • Optimize IngestCtxMap construction {es-pull}120833[#120833]

  • Optimize IngestDocMetadata isAvailable {es-pull}120753[#120753]

  • Optimize IngestDocument FieldPath allocation {es-pull}120573[#120573]

  • Optimize some per-document hot paths in the geoip processor {es-pull}120824[#120824]

  • Returning ignored fields in the simulate ingest API {es-pull}117214[#117214]

  • Run GetPipelineTransportAction on local node {es-pull}120445[#120445]

  • Run TransportGetEnrichPolicyAction on local node {es-pull}121124[#121124]

  • Run template simulation actions on local node {es-pull}120038[#120038]

License

  • Bump TrialLicenseVersion to allow starting new trial on 9.0 {es-pull}120198[#120198]

Logs

  • Add LogsDB option to route on sort fields {es-pull}116687[#116687]

  • Add a new index setting to skip recovery source when synthetic source is enabled {es-pull}114618[#114618]

  • Configure index sorting through index settings for logsdb {es-pull}118968[#118968] (issue: {es-issue}118686[#118686])

  • Optimize loading mappings when determining synthetic source usage and whether host.name can be sorted on. {es-pull}120055[#120055]

Machine Learning

  • Add DeBERTa-V2/V3 tokenizer {es-pull}111852[#111852]

  • Add Inference Unified API for chat completions for OpenAI {es-pull}117589[#117589]

  • Add Jina AI API to do inference for Embedding and Rerank models {es-pull}118652[#118652]

  • Add enterprise license check for Inference API actions {es-pull}119893[#119893]

  • Adding chunking settings to IbmWatsonxService {es-pull}114914[#114914]

  • Adding default endpoint for Elastic Rerank {es-pull}117939[#117939]

  • Adding endpoint creation validation for all task types to remaining services {es-pull}115020[#115020]

  • Check for presence of error object when validating streaming responses from integrations in the inference API {es-pull}118375[#118375]

  • Ignore failures from renormalizing buckets in read-only index {es-pull}118674[#118674]

  • Inference duration and error metrics {es-pull}115876[#115876]

  • Migrate stream to core error parsing {es-pull}120722[#120722]

  • Remove all mentions of eis and gateway and deprecate flags that do {es-pull}116692[#116692]

  • Remove deprecated sort from reindex operation within dataframe analytics procedure {es-pull}117606[#117606]

  • Retry on ClusterBlockException on transform destination index {es-pull}118194[#118194]

  • [Inference API] fix spell words: covertToString to convertToString {es-pull}119922[#119922]

Mapping

  • Add Optional Source Filtering to Source Loaders {es-pull}113827[#113827]

Network

  • Allow http unsafe buffers by default {es-pull}116115[#116115]

  • Http stream activity tracker and exceptions handling {es-pull}119564[#119564]

  • Remove HTTP content copies {es-pull}117303[#117303]

  • ConnectTransportException returns retryable BAD_GATEWAY {es-pull}118681[#118681] (issue: {es-issue}118320[#118320])

Ranking

  • Set default reranker for text similarity reranker to Elastic reranker {es-pull}120551[#120551]

Recovery

  • Allow archive and searchable snapshots indices in N-2 version {es-pull}118941[#118941]

  • Trigger merges after recovery {es-pull}113102[#113102]

Reindex

  • Change Reindexing metrics unit from millis to seconds {es-pull}115721[#115721]

Search

  • Add match support for semantic_text fields {es-pull}117839[#117839]

  • Add support for sparse_vector queries against semantic_text fields {es-pull}118617[#118617]

  • Add support for knn vector queries on semantic_text fields {es-pull}119011[#119011]

  • Adding linear retriever to support weighted sums of sub-retrievers {es-pull}120222[#120222]

  • Address and remove any references of RestApiVersion version 7 {es-pull}117572[#117572]

  • Feat: add a user-configurable timeout parameter to the _resolve/cluster API {es-pull}120542[#120542]

  • Make semantic text part of the text family {es-pull}119792[#119792]

  • Only aggregations require at least one shard request {es-pull}115314[#115314]

  • Prevent data nodes from sending stack traces to coordinator when error_trace=false {es-pull}118266[#118266]

  • Propagate status codes from shard failures appropriately {es-pull}118016[#118016] (issue: {es-issue}118482[#118482])

Security

  • Add refresh .security index call between security migrations {es-pull}114879[#114879]

Snapshot/Restore

  • Add IMDSv2 support to repository-s3 {es-pull}117748[#117748] (issue: {es-issue}105135[#105135])

  • Expose operation and request counts separately in repository stats {es-pull}117530[#117530] (issue: {es-issue}104443[#104443])

  • Retry S3BlobContainer#getRegister on all exceptions {es-pull}114813[#114813]

  • Retry internally when CAS upload is throttled [GCS] {es-pull}120250[#120250] (issue: {es-issue}116546[#116546])

  • Track shard snapshot progress during node shutdown {es-pull}112567[#112567]

Suggesters

  • Extensible Completion Postings Formats {es-pull}111494[#111494]

TSDB

  • Increase field limit for OTel metrics to 10 000 {es-pull}120591[#120591]

Transform

  • Add support for extended_stats {es-pull}120340[#120340]

  • Auto-migrate max_page_search_size {es-pull}119348[#119348]

  • Create upgrade mode {es-pull}117858[#117858]

  • Wait while index is blocked {es-pull}119542[#119542]

  • [Deprecation] Add transform_ids to outdated index {es-pull}120821[#120821]

Vector Search

  • Even better(er) binary quantization {es-pull}117994[#117994]

  • Speed up bit compared with floats or bytes script operations {es-pull}117199[#117199]

New features

CRUD

  • Metrics for indexing failures due to version conflicts {es-pull}119067[#119067]

ES|QL

  • ESQL - Add Match function options {es-pull}120360[#120360]

  • ESQL - Allow full text functions disjunctions for non-full text functions {es-pull}120291[#120291]

  • ESQL: Enable async get to support formatting {es-pull}111104[#111104] (issue: {es-issue}110926[#110926])

  • Expand type compatibility for match function and operator {es-pull}117555[#117555]

  • ST_EXTENT aggregation {es-pull}117451[#117451] (issue: {es-issue}104659[#104659])

  • Support ST_ENVELOPE and related (ST_XMIN, ST_XMAX, ST_YMIN, ST_YMAX) functions {es-pull}116964[#116964] (issue: {es-issue}104875[#104875])

Extract&Transform

  • [Connector API] Support hard deletes with new URL param in delete endpoint {es-pull}120200[#120200]

  • [Connector API] Support soft-deletes of connectors {es-pull}118669[#118669]

  • [Connector APIs] Enforce index prefix for managed connectors {es-pull}117778[#117778]

Highlighting

  • Add Highlighter for Semantic Text Fields {es-pull}118064[#118064]

Infra/Core

  • Infrastructure for assuming cluster features in the next major version {es-pull}118143[#118143]

Machine Learning

  • ES|QL categorize with multiple groupings {es-pull}118173[#118173]

  • Support mTLS for the Elastic Inference Service integration inside the inference API {es-pull}119679[#119679]

  • [Inference API] Add node-local rate limiting for the inference API {es-pull}120400[#120400]

Mapping

  • Add option to store sparse_vector outside _source {es-pull}117917[#117917]

Ranking

  • Add a generic rescorer retriever based on the search request’s rescore functionality {es-pull}118585[#118585] (issue: {es-issue}118327[#118327])

Relevance

  • Add Multi-Field Support for Semantic Text Fields {es-pull}120128[#120128]

Vector Search

  • Add new experimental rank_vectors mapping for late-interaction second order ranking {es-pull}118804[#118804]

  • KNN vector rescoring for quantized vectors {es-pull}116663[#116663]

  • Mark bbq indices as GA and add rolling upgrade integration tests {es-pull}121105[#121105]

Upgrades

Infra/Core

  • Bump major version for feature migration system indices {es-pull}117243[#117243]

  • Update ASM 9.7 → 9.7.1 to support JDK 24 {es-pull}118094[#118094]

Machine Learning

  • Automatically rollover legacy .ml-anomalies indices {es-pull}120913[#120913]

  • Automatically rollover legacy ml indices {es-pull}120405[#120405]

  • Change the auditor to write via an alias {es-pull}120064[#120064]

Search

  • Upgrade to Lucene 10 {es-pull}114741[#114741]

  • Upgrade to Lucene 10.1.0 {es-pull}119308[#119308]