Add option to override template based on a hash

[ycombinator]

Problem

Often I will have a geo point field in my data that I’m indexing into Elasticsearch from Logstash, perhaps something like:

"location": {
    "lat": 51.5034070,
    "lon": -0.1275920
}

In order to give this field the correct mapping, I end up cloning https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json and appending a new location property with the appropriate geo_point mapping to the properties object at the bottom of the template.

Proposal

Consider providing a top-level geo_point field in the template, similar to the geoip.location field, with the appropriate geo_point mapping

Pros

IMHO, this is a better experience for users who may have a geo point field in there data. They no longer need to clone and modify the template just to add the correct mapping for such a field.

Cons

By providing yet-another field in the template properties, we're increasing the chances that a field with the same name already exists in the user's data, but as a non-geo-point field. For example, if we named our field in the template as location and the user already had data like "location": "London, UK" we would be screwing up their expected mapping. We could possibly alleviate this by naming the field something special like '@location', which would also be consistent with the other @-fields in Logstash.

Alternatives

Instead of hardcoding a top-level @location field in the template properties, we could add a dynamic template like so:

{
  "location_geo_point_fields" : {
    "match" : "@location*",
    "match_mapping_type" : "geo_point",
    "mapping" : { "type" : "geo_point", "doc_values" : true }
  }
}

This way the user could specify multiple geo point fields in their data. As long as they started with @location, the correct mapping would be applied.

[andrewvc]

@ycombinator thanks for the excellent writeup! It is a tricky thing here. Who knows what people are already stuffing in @location?

In my opinion we should find a way to solve the mapping problem a little better. Perhaps it would be best to allow for dynamic mapping configuration in the plugin config itself? Maybe you could do something like:

elasticsearch {
  hosts => ["localhost:9200"]
  mapping_properties => {
    location => { type => geo_point }
  }
}

What do you think?

[jamesmanning]

@andrewvc while it supports the string and array forms as well, is it possible to have the mapping to geo_point "just work" with @location (or as the dynamic template mapping) as long as it is an object with 'lat' and 'lon' properties?

While I could see @location (or @location*) being something possibly used by existing users, if it's an object with those properties, it would seem to be a safe (IMHO) assumption that it should be mapped as a geo_point.

True, it's not as convenient as also accepting the string and array formats, and that'd be great to also support those, but at least it gives people an option to get geo_point mapping without having to modify their template.

@ycombinator do you think this is a potential reasonable compromise?

Thanks!

[jsvd]

@ycombinator one and a half years later, any new ideas on this? :)

[ycombinator]

@jsvd Oops, sorry for dropping the ball on this and thanks for the ping.

I like @andrewvc's suggestion. Perhaps we could generalize that setting even further and call it something like override_template. The ES output would then take the value of that setting and overlay it on the default template.

WDYT?

[jsvd]

I'm +1 on exploring an option to have an override option that takes a hash and merges that with the template.
I think there may be some challenged because of the way the logstash dsl works, but we can validate that during implementation.

[jsvd]

with this in mind, can you change the issue name to something like "add an option to override the template based on a hash"? (and label this an enhancement)

[ycombinator]

Does the override_template option have to be a hash? Could it be a path to a JSON file instead?

Also, since we have an option called template_overwrite, it might be confusing to call the proposed option override_template. Maybe we call it overlay_template or merge_template or something like that?

[jsvd]

Maybe we call it overlay_template or merge_template or something like that?

right, I wasn't thinking too much of the name itself, just that it makes sense to implement the feature.

Does the override_template option have to be a hash? Could it be a path to a JSON file instead?

If we get to a point where the option points to a json file, I don't think there's too much difference from just the normal template_path option that takes the path of a json file to be used as the template.

I see the benefit of this new option as being able to tweak the template without needing extra artifacts, which we know complicates the central management use case.