Skip to content

Added YAML configuration examples for tail input plugin docs. #1589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 18, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 61 additions & 4 deletions pipeline/inputs/tail.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,10 +136,10 @@ In your main configuration file, append the following `Input` and `Output` secti

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
```text
[INPUT]
Name tail
Path /var/log/syslog
Name tail
Path /var/log/syslog

[OUTPUT]
Name stdout
Expand Down Expand Up @@ -180,6 +180,9 @@ We need to specify a `Parser_Firstline` parameter that matches the first line of

In the case above we can use the following parser, that extracts the Time as `time` and the remaining portion of the multiline as `log`


{% tabs %}
{% tab title="fluent-bit.conf" %}
```text
[PARSER]
Name multiline
Expand All @@ -188,9 +191,24 @@ In the case above we can use the following parser, that extracts the Time as `ti
Time_Key time
Time_Format %b %d %H:%M:%S
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml
parsers:
- name: multiline
format: regex
regex: '/(?<time>[A-Za-z]+ \d+ \d+\:\d+\:\d+)(?<message>.*)/'
time_key: time
time_format: '%b %d %H:%M:%S'
```
{% endtab %}
{% endtabs %}

If we want to further parse the entire event we can add additional parsers with `Parser_N` where N is an integer. The final Fluent Bit configuration looks like the following:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```text
# Note this is generally added to parsers.conf and referenced in [SERVICE]
[PARSER]
Expand All @@ -210,6 +228,31 @@ If we want to further parse the entire event we can add additional parsers with
Name stdout
Match *
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml
parsers:
- name: multiline
format: regex
regex: '/(?<time>[A-Za-z]+ \d+ \d+\:\d+\:\d+)(?<message>.*)/'
time_key: time
time_format: '%b %d %H:%M:%S'

pipeline:
inputs:
- name: tail
multiline: on
read_from_head: true
parser_firstline: multiline
path: /var/log/java.log

outputs:
- name: stdout
match: '*'
```
{% endtab %}
{% endtabs %}

Our output will be as follows.

Expand Down Expand Up @@ -262,12 +305,26 @@ Fluent Bit keep the state or checkpoint of each file through using a SQLite data

The SQLite journaling mode enabled is `Write Ahead Log` or `WAL`. This allows to improve performance of read and write operations to disk. When enabled, you will see in your file system additional files being created, consider the following configuration statement:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```text
[INPUT]
name tail
path /var/log/containers/*.log
db test.db
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml
pipeline:
inputs:
- name: tail
path: /var/log/containers/*.log
db: test.db
```
{% endtab %}
{% endtabs %}

The above configuration enables a database file called `test.db` and in the same path for that file SQLite will create two additional files:

Expand All @@ -284,4 +341,4 @@ The `WAL` mechanism give us higher performance but also might increase the memor

File rotation is properly handled, including logrotate's _copytruncate_ mode.

Note that the `Path` patterns **cannot** match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records.
Note that the `Path` patterns **cannot** match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records.