Describe the bug
Hello,
We are using fluentd as our central log collector component of our project in a containerized fashion. Basically fluentd collects logs from various input sources and outputs them to our postgresql database via postgres output plugin.
Recently, we came across a situation where a buffer file is flushed over and over and never gets expired resulting quite number of duplicated log records in the database (for a case, two lines of logs reached 30000+ duplicated records).
To Reproduce
N/A (System is being tested and at it's not clear which point exactly it came into this situation)
Expected behavior
Flushed buffer should be expired.
Your Environment
- Fluentd version: (Docker image) fluent/fluentd:v1.16.6-debian-1.0
Your Configuration
Buffer section of postgres output plugin:
<buffer time>
@type file
path /fluentd/buffer/sinecnsp_diagnostic_event
flush_thread_count 8
total_limit_size 16GB
flush_mode interval
flush_interval 15s
retry_max_times 8
timekey 300
timekey_wait 30
timekey_use_utc true
flush_at_shutdown true
</buffer>Your Error Log
Additional context
We backed up the mentioned .log file and .log.meta file and if I put them at the buffer path of different setup, issue arises on the new setup also.
For the sake of info security, we cannot share the .log file itself, if it's going to help, I can share the .log.meta file.
I shared only buffer section of postgres output plugin because our configuration is split into quite number of different configuration files and I think only this buffer section is relevant. If you need, I can share from other config files also.
Describe the bug
Hello,
We are using fluentd as our central log collector component of our project in a containerized fashion. Basically fluentd collects logs from various input sources and outputs them to our postgresql database via postgres output plugin.
Recently, we came across a situation where a buffer file is flushed over and over and never gets expired resulting quite number of duplicated log records in the database (for a case, two lines of logs reached 30000+ duplicated records).
To Reproduce
N/A (System is being tested and at it's not clear which point exactly it came into this situation)
Expected behavior
Flushed buffer should be expired.
Your Environment
Your Configuration
Buffer section of postgres output plugin: <buffer time> @type file path /fluentd/buffer/sinecnsp_diagnostic_event flush_thread_count 8 total_limit_size 16GB flush_mode interval flush_interval 15s retry_max_times 8 timekey 300 timekey_wait 30 timekey_use_utc true flush_at_shutdown true </buffer>Your Error Log
Additional context
We backed up the mentioned .log file and .log.meta file and if I put them at the buffer path of different setup, issue arises on the new setup also.
For the sake of info security, we cannot share the .log file itself, if it's going to help, I can share the .log.meta file.
I shared only buffer section of postgres output plugin because our configuration is split into quite number of different configuration files and I think only this buffer section is relevant. If you need, I can share from other config files also.