engine: fix high memory usage in FluentLogEventRouter

[Watson1978]

Which issue(s) this PR fixes:
Fixes #4994

What this PR does / why we need it:
Previously, FluentLogEventRouter was designed to start processing only after the worker had started.

As a result, if a large number of warnings were emitted by the in_tail plugin immediately after the worker started,
lots of warning messages would accumulate in the FluentLogEventRouter's queue.
And it consumes a significant amount of memory.

This PR changes the behavior so that FluentLogEventRouter starts processing before the worker is fully started, allowing it to consume the message queue immediately.

With this change, memory usage is significantly reduced compared to the previous implementation.

Here is the memory usage was obtained using the same operation as #4994.

FYI)
You can confirm the memory usage easily with the following sample code.

queue = Thread::Queue.new

500_000.times do |i|
  queue << "a" * 1024
end

while !queue.empty?
  queue.pop
end

GC.start

rss = Integer(`ps -o rss= -p #{Process.pid}`) / 1024.0
puts "Memory Usage: #{rss} MB"

$ ruby t.rb
Memory Usage: 539.25 MB

Docs Changes:

Release Note:

[daipom]

Thanks!
I see. Actually, in_tail starts reading logs before completing start phase unless setting skip_refresh_on_startup.
(This may not be preferable...)

However, it appears to me that we have to be very careful about changing this starting order.
Since FluentLogEventRouter is related to event routing, it would be safe to start it after all plugins have been started.

For example, it would cause a bug if events are emitted without some plugins started.
(in_tail is doing this, but fortunately, the input plugins start last, so it likely hasn't caused any issues.)

[ashie]

It might be better to implement #1504 instead of this change.
Running refresh_watchers for huge amount of logs at start phase causes any other problems too, for example it blocks starting process of any other plugins & fluentd itself until finishing it.

[Watson1978]

it would be safe to start it after all plugins have been started.

I can't understand that the current is safe because it is already known to cause memory leaks.

[daipom]

it would be safe to start it after all plugins have been started.

I can't understand that the current is safe because it is already known to cause memory leaks.

I don't mean to say the current code is entirely safe.
I mean that starting FluentLogEventRouter after all plugins have been started is safe for the event routing process.

[daipom]

It might be better to implement #1504 instead of this change. Running refresh_watchers for huge amount of logs at start phase causes any other problems too, for example it blocks starting process of any other plugins & fluentd itself until finishing it.

I see!
It would not be a fundamental solution, but it would be very beneficial!

[Watson1978]

For example, it would cause a bug if events are emitted without some plugins started.

I think FluentLogEventRouter handles log routing.
The log should be generated because the plugin is running, and if it is not, it should be fixed.

[Watson1978]

We need another approach to fix this