2.0.0-rc1

Release Candidate

This is the release candidate for NEST 2.x! Many improvements and fixes went into this release, including providing support for two new target frameworks, dotnet5.1 (.NETPlatform5.1) and net46 (.NETFramework4.6). As always, your feedback on this release are very welcome!

Improvements

• #1697 SUPPORT .NET CORE WITH DNX TOOLING
• #1089 Refactor to use System.Linq.Expressions.ExpressionVisitor
• #1686 Support other_bucket and other_bucket_key in filters aggregations
• #1705 AutoMap/Infer TimeSpan properties as long ticks
• #1742 Support looping of named aggregation buckets
• #1755 Add .NET 4.6 as a target framework

Bug Fixes

• #985 Reindex supports _parent and _routing
• #1433 Reindex supports reindexing one/many/all types within the from index
• #1490 Update now supports fields and is not generic
• #1612 Unify the methods and overloads available on Query<T>
• #1634 Ensure consistent purified Uris across all supported platforms
• #1708 Property Name should resolve to the last token
• #1729 Add Aggregation Metadata
• #1731 Use custom JsonConverters when supplied on derive JsonNetSerializer
• #1763 Add _timestamp to IHit<T>
• #1763 Add _ttl to IHit<T>
• #1766 when passed an a null reference for string, return null reference for DateMath
• #1767 Fields should not be null on response when specified on request
• #1772 Fix deficiencies in the dnx tooling for supporting dependency version ranges
• #1778 Align request/response type names
• #1776 Rename types in line with Naming Conventions
• #1773 Make ConnectionSettings protected on JsonNetSerializer

2.0.0-alpha2

Bug Fixes

• #1694 Add support for Newtosoft.Json 8
• #1706 Separate types for histogram and date histogram items
• #1709 Boolean and "conditionless" queries
• #1712 Rename ConnectionStatusHandler to OnRequestCompleted and add back functionality
• #1713 High level methods now take Time instead of TimeSpan
• #1720 SniffingConnectionPool now implements IDisposable (ty @garymcleanhall)
• #1726 Improved Norms implementation
• #1728 Add support for range queries on string fields
• #1736 Fix choosing the correct HTTP method when indexing an object with no Id
• #1741 Fix snowball analyzer language case sensitivity

2.0.0-alpha1

This marks the first release of our 2.0 branch (now master) with well over a 1000 commits since 1.7.1 (the currently last released NEST version in the 1.x range).

We are releasing it as alpha because we currently only support .NET 4.5 and up. For us to go from alpha to beta we ideally still have to finish

• PCL/.NET CORE are in the works but not part of our build and CI infrastructure.
• Finish our new documentation

Please note that even though this is an alpha, we have fully tested this against Elasticsearch 2.0 up to 2.1 (the current latest version).

Back to the drawing board

We took sometime to go back to the drawing board. NEST is quite old (started in 2010) and not all of the choices that have accumulated in the code base make sense anymore.
So we stepped back, formulized properly how we see the lifetime of a call, and worked off of that. Armed with the below diagram, we completely rewrote NEST's internals. The old TPL based code base is now replaced with async/await, we have a much saner approach to exceptions and errors, and we expose enough information as an audit trail so you do not have to ever guess what went down during a call.

Our internals now also reflect this:

IElasticClient exposes all the Elasticsearch API endpoints e.g client.Search this calls into ITransport's 2 methods Request and RequestAsync the default ITransport then uses the passed in IRequestPipelineFactory to create an RequestPipeline which implements IPipeline.

This pipeline now handles all of the failover/sniffing/pinging logic and directly reflects the flow diagram.

We also simplified IConnection down just 2 methods. This means the outer edges are clean (ITransport and IConnection) and implementing your own should be really really simple. All of these (and also IMemoryStreamProvider and IDateTimeProvider) can be injected on the constructor of the client.

Test Framework

Another huge endeavour is the rework of our test framework. NEST 1.x was always well tested but used 5 different test projects and 5 years worth of changing our minds as to how best to write tests and assertions.
Thus becoming a big hodgepodge of nunit assertions, fluent assertions, FakeItEasy, Moq combined with several different ways to compare json with object graphs and vice-versa. Trying to write a new test quickly became cumbersome because there was no clear cut way how best to write said test.

So the first thing we did as part of our 2.0 branch was to completely delete all of our tests. This act of insanity gave us carte blanche during our rewrite.

As of 2.0 we have one test project Tests and all of the tests are written in such a way that they can be run in unit test mode and integration test mode. Write once, run differently.
All the API endpoint tests test all 4 variations, 2 DSL's (fluent and object initializer) + sync and async. We also test all of the moving parts of Elasticsearch DSL (Aggregations, Sorting, IndexSettings, etc...) in the same way.

We also introduced a thing we dubbed Literate Testing to allow us to write tests in a more story telling form with the comments serving as the asciidoc for our documentation while using the Roslyn compiler to pick the interesting bits of code.
This gives us the benefit of always compiling our documentation but also having one place where we document, test and assert how a piece of code is supposed to work.

Another huge component of our testing framework is the Virtual Cluster that allows us to write tests for any situation and how we expect the client to behave.

/** we set up a 10 node cluster with a global request time out of 20 seconds.
* Each call on a node takes 10 seconds. So we can only try this call on 2 nodes
* before the max request time out kills the client call.
*/
var audit = new Auditor(() => Framework.Cluster
  .Nodes(10)
  .ClientCalls(r => r.FailAlways().Takes(TimeSpan.FromSeconds(10)))
  .ClientCalls(r => r.OnPort(9209).SucceedAlways())
  .StaticConnectionPool()
  .Settings(s => s.DisablePing().RequestTimeout(TimeSpan.FromSeconds(20)))
);

audit = await audit.TraceCalls(
  new ClientCall {
    { BadResponse, 9200 }, //10 seconds
    { BadResponse, 9201 }, //20 seconds
    { MaxTimeoutReached }
  },
  /**
  * On the second client call we specify a request timeout override to 80 seconds
  * We should now see more nodes being tried.
  */
  new ClientCall(r => r.RequestTimeout(TimeSpan.FromSeconds(80)))
  {
    { BadResponse, 9203 }, //10 seconds
    { BadResponse, 9204 }, //20 seconds
    { BadResponse, 9205 }, //30 seconds
    { BadResponse, 9206 }, //40 seconds
    { BadResponse, 9207 }, //50 seconds
    { BadResponse, 9208 }, //60 seconds
    { HealthyResponse, 9209 },
  }
);

This showcases the Virtual Cluster tests combined with Literate Tests and the extensive audit trail information available on each response (or exception).

I'm pleased to say we are back at a decent coverage rate (60%) and we'll continue to bump that.

Exception handling

Another big change in NEST 2.0 is how we deal with exceptions.

In NEST 1.x, the client threw a multitude of exceptions: MaxRetryException, ElasticsearchAuthException, ElasticsearchServerException, DslException, etc.. This made it challenging for users to handle exceptions/invalid responses, and understand the root cause of errors. On top of that, it depended on what kind of IConnectionPool was injected to contain maximum backwards compaitibility with NEST 0.x

In NEST 2.x, exceptions are much more deterministic. The former ThrowOnElasticsearchServerExceptions() setting has been replaced with simply ThrowExceptions(), which determinse whether the client should ever throw an exception or not (client side and server exceptions). Furthermore, the types of exceptions have been reduced and simplified-the client will now only throw three types of exceptions:

ElasticsearchClientException: These are known exceptions, either an exception that occurred in the request pipeline(such as max retries or timeout reached, bad authentication, etc...) or Elasticsearch itself returned an error (could not parse the request, bad query, missing field, etc...). If it is an Elasticsearch error, the ServerError property on the response will contain the the actual error that was returned. The inner exception will always contain the root causing exception.

UnexpectedElasticsearchClientException: These are unknown exceptions, for instance a response from Elasticsearch not properly deserialized. These are usually bugs and should be reported. This exception also inherits from ElasticsearchClientException so an additional catch block isn't necessary, but can be helpful in distinguishing between the two.

Development time exceptions: These are CLR exceptions like ArgumentException, NullArgumentException etc., that are thrown when an API in the client is misused. These should not be handled as you want to know about them during development.

Breaking Changes

Even though a lot of work went into the interior, the exterior did not escape unscathed! On top of the many breaking changes that Elasticsearch 2.0 introduces, there are more then a few NEST 2.0 introduces.
We revalidated all the request and response domain objects against Elasticsearch 2.0.

We will do our best to compile a list when NEST 2.0 GA hits. If we moved your cheese to a spot you can no longer find it then please open an issue and we'll be more then happy to help locate it.

Elasticsearch 2.x support

NEST 2.0 supports all the new features from Elasticsearch 2.0 including pipeline aggregations. Here we'll just highlight a couple features that are reflected in NEST changes

Removal of filters

NEST 2.0 reflects Elasticsearch 2.0 move of filters and no longer has filter constructs in its code base

Filtered query deprecation

With the removal of filters NEST has added a special construct in its Query DSL to easily create a bool query with a filter clause

.Query(q=> +q.Term(p=>p.Name, "NEST"))

Note the + this will wrap the term query inside a bool query's filter clause

You can even combine this with !

.Query(q=> !+q.Term(p=>p.Name, "NEST"))

This will wrap the term query inside a bool filter and that bool inside a bool must_not, obviously this also works for the object initializer syntax

!+new TermQuery {}

Attribute based mapping

The single ElasticPropertyAttribute has been broken up into individual attributes per property type.

For instance, the following:

[ElasticType(Name = "othername", IdProperty = "MyId")]
public class Foo
{
  [ElasticProperty(Type = FieldType.String)]
  public Guid MyId { get; set; }

  [ElasticProperty(Type = FieldType.String)]
  public string Name { get; set; }

  [ElasticProperty(Type = FieldType.String, Analyzer = "myanalyzer", TermVector = TermVectorOption.WithOffsets)]
  public string Description { get; set; }

  [ElasticProperty(Type = FieldType.Date, Format = "mmmddyyyy")]
  public DateTime Date { get; set; }

  [ElasticProperty(Type = FieldType.Integer, Coerce = true)]
  public int Number { get; set; }

  [ElasticProperty(Type = FieldType.Nested, IncludeInParent = true)]
  public List<Bar> Bars { get; set; }
}

becomes

[ElasticsearchType(Name = "othername", IdProperty = "MyId")]
public class Foo
{
  [String]
  public Guid MyId { get; set; }

  [String]
  public string Name { get; set; }

  [String(Analyzer = "myanalyzer", TermVector = TermVectorOption.WithOffsets)]
  public string Description { get; set; }

  [Date(Format = "mmddyyyy")]
  public DateTime Date { get; set; }

  [Number(NumberType.Integer, Coerce = true, DocValues = true)]
  public int Number { get; set; }

  [Nested(IncludeInParent = true)]
  public...

1.7.1

Bug Fixes

• #1533 Added missing store option to _timestamp field (ty @KodrAus !)
• #1525 Added missing offset option to date_histogram aggregation
• #1514 Added missing filter option to has_parent filter
• #1505 Added missing min and max options to has_child filter
• #1497 Fixed serialization of fields on UpdateDescriptor when using expressions

1.7.0

This release contains full feature parity with Elasticsearch 1.7.

New Features

• #1381 MapFromAttributes() now defaults Guids to strings
• #1454 Add Query(IQueryContainer) to CountDescriptor
• #1479 byte and short types are now mapped to their respective Elasticsearch types when using MapFromAttributes()
• #1485 Add support for version to snapshot info
• #1487 Add shard_size to TermStats face (TY @qbast)

Bug Fixes

• #1296 UpdateReponse was missing Fields
• #1440 Better syntax for ReIndex
• #1474 GeoBounds aggregation not read to completion
• #1478 Fixed serialization of LanguageAnalyzers stem_exclusion property
• #1484 Document metadata missing on TermVectorResponse
• #1496 Force invariant culture on all dates

1.6.1

• NEST upgraded to Json.NET 7.x
• #1463 Fix ScriptFile and ScriptId not being serialized on bulk update requests
• e344795 Fix boost option missing from several queries

1.6.0

This release contains full feature parity with Elasticsearch 1.6.

New Features

• #1446 Support for field stats API
• #1445 Support for synced flush API
• #1444 Support for response filtering on all APIs
• #1441 Support for murmur3 Field Type
• #1430 Support for include/exclude on TermsAggregation
• #1451 Added node id to _cat API responses
• #1450 Added default option to FieldValueFactor
• #1449 Added numeric_resolution to date mapping
• #1447 Added explanation to validate API response
• $1448 Added SortMulti() to allow for easy sorting across multiple fields (TY @mgoodfellow !)

Bug Fixes

• #1459, #1456 Add support for script, script id, and script file where they were missing on all update descriptors
• #1457 Fixed NullReferenceException in GetIndex() when a custom analyzer had no type defined
• #1438 GeoPoint mapping was missing precision
• #1429 DateMapping, GeoPointMapping, and BooleanMapping were missing doc_values
• #1409 Raw filters were not written to request when used on filtered aliases
• #1452 Fixed serialization of KeepWordsPath property on KeepWordsTokenFilter (TY @davidtme !)
• #1431 MoreLikeThis query should accept documents or ids (TY @robertlyson !)

1.5.1

Bug Fixes

• #1310 Suggest<T> doesn't infer index name from T(TY @gmoskovicz)
• #1323 MultiPercolate now has support for PercolateMany (TY @gmoskovicz)
• #1401 Keep exceptions stack trace in async calls (TY @vbfox)
• #1403 Multiple Extended Stats aggregations are not parsed correctly in ISearchResponse
• #1407 Support distance_type for SortGeoDistance
• #1410 Added overload to DismaxQueryDescriptor's Queries method to take QueryContainer[](TY @xueye)
• #1412 RangeFilterJsonConverter doesn't correctly serialize gt and lt (TY @aochsner)
• #1416 IMultiGetOperation can now be deserialized (TY @abibell)
• #1418 function_score query doesn't have filter property (TY @gmoskovicz)

1.5.0

This is the 1.5 release! This release contains full feature parity with Elasticsearch 1.5. As always, thank you to everyone who contributed!

Also, checkout our new tutorial repository that will guide you through building a search interface for NuGet packages. It's a great way to get started with NEST and Elasticsearch.

New Features

• #1369 INNER HITS SUPPORT !
• We now support lazy deserialization of documents using IDocument (side-effect of implementing inner hits)
• Low-level client now up to date with 1.5 rest spec
• #998 You can now use variables in .Suffix() in property path Expressions
• #1387 Fielddata fields support on Search()
• #1316 Added UsePrettyRequests() and PrettyJson() to Connectionsettings
• #1295 Added script_file to transform mapping (ty @markuslindberg!)
• #1300 Added an internal elasticsearch timespan string parser (ty @aochsner!)
• #1305 Special .ConditionlessWhen(true_expression) to functionscore queries to control its conditionless nature separately
• #1325 Included index property for Indices Filter ty @abibell!
• #1307 IncludeInParent support in attribute based mappings ty @robertlyson!
• #1333 Missing fuzzy_transpositions support in MatchQuery ty @robertlyson!
• #1368 Added percentage and scripted heuristics to significant terms
• #1362 Support for supplying index settings on restore ty @robertlyson!
• #1376 Added status and total memory to cat indices response
• #1341 MLT query: Support for artificial documents
• #1354 MLT query: Support for minimum_should_match parameter
• #1109 MLT query: Support for per_field_analyzer
• #1373 Support for min_score parameter on function score query ty @robertlyson!
• #1342 Added timezone to query_string
• #1343 Added format to range filter and query
• #1374 Support for cat segments API
• #1356 Support for minimum_should_match option on query_string_query ty @robertlyson!
• #1388 Has child/top children: support for min score type and min/max_children
• #1105 Added default and ignore_missing to timestamp mappings
• #1350 Added new columns in the cat nodes API
• #1352 Added support for _field_names mapping
• #1357 Added support for analyze_wildcard on simeple_query_string
• #1358 Added max_determinized_states to regexp filter and query and query string query
• #1363 Added more translog stats on recovery api and cat recovery
• #1364 Added pending tasks to cluster health and cat health API

Bug Fixes

• #1347 Fix bad nuspec output path for .NET 4.0 nuget packages ty @cstlaurent!!
• #1336 Fix id property lookup methods configured on connection settings not being cached
• #1292 Allow sort() on fluent search descriptor no longer throws when passed null
• Script sort should not be conditionless when only file is set
• #1297 Conditonless queries inside dismax_query wre serialized as {} ty @danielsilva!
• #1306 Fixed some several typo's in links in the documentation ty @romansp!
• #1317 Fieldvalue helpers on response should not throw if field is missing
• #993 and #992 Better documented some of the breaking changes from NEST 0.* to 1.*
• Fixed typo in SnapshotBytesPerSecondMaximum() reported by @shanekenney 7a52b89

1.4.3

This 1.4.3 release is compatible with all Elasticsearch 1.x releases.
The previous 1.4.2 release is compatible up to (but not including) Elasticsearch 1.5.

Bug Fixes

• #1291 Fix broken aggregations on Elasticsearch 1.5 due to elastic/elasticsearch#10284
• #901 Fix enums from being skipped when calling MapFromAttributes(). They are now mapped as integers by default.
• #1254 Fixed exception occurring when using Thrift and opening many simultaneous async connections (Ty @lukapor !)
• #1281 Added support for standard deviation bounds to the extended stats aggregation response
• #1279 Fixed an ArgumentNullException that occurred when using the MultiSearch object initializer syntax
• #1275 Aliases was missing from cluster state response

Enhancements

• #1267 Improved tracing: requests are now traced when they begin and end
• #1280 Added a convenience extension method for passing a template name directly to TemplateExists() rather than using descriptors