Elapsed filter with load balanced logstash in Filebeat

[ivasilyev-servicetitan-com]

Hello.

I have Filebeat with 4 logstash servers configured with load balancing on

  logstash:
    # The Logstash hosts
    hosts: ["logstash1", "logstash2", "logstash3", "logstash4"]

    # Optional load balance the events between the Logstash hosts
    loadbalance: true

And I have logstash-filter-elapsed configured on my logstash servers.

The problem is when "start" event comes to one logstash instance, and "end" event comes to another logstash server I'm getting elapsed_end_without_start with no elapsed_time calculated (obviously).

I didn't find any notes about dealing with this filter in loadbalanced logstash env.
Are there any recommendations how this can be configured?
Or this plug in shouldn't be used in loadbalanced env? Then it makes sense to add this info into documentation.

Thanks.

[sc7565]

We are also facing same issue? did you get solution for this, as for us we decided to append the Logstash hostname to unique_id_field so it is unique for each LS

[ivasilyev-servicetitan-com]

I didn't find the solution, was going to disregard of Elapsed filter in favour to custom solution.
But how does "append LS hostname to unique_id_field" help you? Can you please explain.

Cause how I understand this:

• Filebeat sends start_event log to LS1
• Filebeat sends end_event log to LS2

So anyway my end_event will have elapsed_end_without_start. How modifying unique_id_field can help to find start_event log on another LS1 server?

[sc7565]

Sorry for confusion, in our case we are using only start_event and timeout where we want to reset count to 0 on that logstash and finally clone event and send it to ES so other LS can read from there to get current count. we are also using Kafka as message broker and hence we have no control over which event goes where but as we use unique field as combination with hostname the elapsed filter calculates time correctly on that LS.