Skip to content

administration: buffering-and-storage: correct chunks up indication #2098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 7, 2025
Merged

administration: buffering-and-storage: correct chunks up indication #2098

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a8a5085
administration: buffering-and-storage: fix typo in mem_buf_limit
lecaros Oct 7, 2025
481bc6e
administration: buffering-and-storage: add missing ')'s
lecaros Oct 7, 2025
5341405
administration: buffering-and-storage: fix chunks up indication
lecaros Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions administration/buffering-and-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ If this input uses more than 50 MB memory to buffer logs, you will get a wa

{% hint style="info" %}

`m em_buf_Limit` applies only when `storage.type` is set to the default value of `memory`.
`mem_buf_Limit` applies only when `storage.type` is set to the default value of `memory`.

{% endhint %}

Expand All @@ -103,7 +103,7 @@ Enabling filesystem buffering changes the behavior of the engine. Upon chunk cre

Fluent Bit controls the number of chunks that are `up` in memory by using the filesystem buffering mechanism to deal with high memory usage and backpressure.

By default, the engine allows a total of 128 chunks `up` in memory in total, considering all chunks. This value is controlled by the service property `storage.max_chunks_up`. The active chunks that are `up` are ready for delivery and are still receiving records. Any other remaining chunk is in a `down` state, which means that it's only in the filesystem and won't be `up` in memory unless it's ready to be delivered. Chunks are never much larger than 2 MB, so with the default `storage.max_chunks_up` value of 128, each input is limited to roughly 256 MB of memory.
By default, the engine allows a total of 128 chunks `up` in memory in total, considering all chunks. This value is controlled by the service property `storage.max_chunks_up`. The active chunks that are `up` are either ready for delivery (marked busy and locked), or are still receiving records. Any other remaining chunk is in a `down` state, which means that it's only in the filesystem and won't be `up` in memory unless it's ready to be delivered. Chunks are never much larger than 2 MB, so with the default `storage.max_chunks_up` value of 128, each input is limited to roughly 256 MB of memory.

If the input plugin has enabled `storage.type` as `filesystem`, when reaching the `storage.max_chunks_up` threshold, instead of the plugin being paused, all new data will go to chunks that are `down` in the filesystem. This lets you control memory usage by the service and also provides a guarantee that the service won't lose any data. By default, the enforcement of the `storage.max_chunks_up` limit is best-effort. Fluent Bit can only append new data to chunks that are `up`. When the limit is reached chunks will be temporarily brought `up` in memory to ingest new data, and then put to a `down` state afterwards. In general, Fluent Bit works to keep the total number of `up` chunks at or under `storage.max_chunks_up`.

Expand All @@ -112,8 +112,8 @@ If `storage.pause_on_chunks_overlimit` is enabled (default is off), the input pl
Look for messages in the Fluent Bit log output like:

```text
[input] tail.1 paused (storage buf overlimit
[input] tail.1 resume (storage buf overlimit
[input] tail.1 paused (storage buf overlimit)
[input] tail.1 resume (storage buf overlimit)
```

##### Limiting filesystem space for chunks
Expand Down