Filter randomly drops events

[michaeleino]

using this filter is really great,
but i have noticed that it drops some events, & this is randomly I see the end tasks with the tag "elapsed_end_without_start".

I have tried to set the pipeline workers to 1, but it didn't help... just degrade the performance to the ground.
my system config:
Ubuntu 16.04
logstash 5.3.2
logstash-filter-elapsed (4.0.1)
Elasticsearch 5.3.1

elapsed {
periodic_flush => true
start_tag => "startevent"
end_tag => "endevent"
unique_id_field => "ID"
timeout => 600
new_event_on_match => false
add_tag => [ "autoelapsed" ]
}

[melbouhy]

Hello,

Did you find ans solution ? Because I have the exact same problem .

Thanks

[michaeleino]

seems no solution for elapsed plugin... adding it takes much CPU resource

I didn't use it anymore... i can calculate this duration by scripted fields in kibana

[melbouhy]

@michaeleino Could you please give me an example how are you doing this ? Which Kibana version are you using ?

[michaeleino]

from Kibana... go to managment > Index Patterns > Scripted fields
Add new field choose it to be painless & use this simple script

!doc['End_Time'].empty ? ((doc['End_Time'].value - doc['Start_Time'].value )/1000) : ''

replace the "End_Time/Start_Time" with your field names.

equation will output in milliseconds, I use /1000 to let the output in seconds

But this will lack the feature of directly calculate & aggregate on this field outside kibana, you will need to calculate it in each search.

[sivachandragit]

Hi,

I have only one field called timestamp and I want to know the processing time of each request which means time difference between 2 documents which contains a unique identifier.

[DirkRichter]

It seems, that tag "elapsed_end_without_start" occurs with multiple pipeline workers. Try to isolate the elapse-computation into it's own pipeline and use pipeline.workers=1 for this pipeline.

[gomclifton]

I have this issue also, after changing the pipeline.workers=1, it helped reduce a lot of the elapsed_end_without_start, but we still get a small # of the errors. Is there any more work being done on this?

[gomclifton]

Are there any plans to fix the elapsed_end_without_start errors? I've also reduced pipeline.workers to 1, but these errors still occur.

[gvorobyo]

still actual, same issue for me