Add an option to control the maximum size of JSON output

[LionelCons]

When using an output plugin like http to send JSON data, one can mainly choose between the json_lines and json formats.

With json_lines, every log entry triggers a POST request, which implies a significant overhead, both on the network and on the receiving end.

With json, a single POST request can group multiple log entries. This is good. However, if there are many entries to send, this can result in sending a huge POST request that the receiver might struggle to handle.

It would be useful to be able to control the maximum size of the JSON output. It could be a maximum number of log entries or a maximum number of bytes.

[edsiper]

@LionelCons

On every flush call (applicable for all output plugins), they receive a Chunk which contains N records. In the case of the HTTP output plugin, the json_lines format just compose the payload in a different way than json, adding a breaking \n between each record. Note: there is a single HTTP POST to the remote end-point containing all the records together, not one per record.

[LionelCons]

@edsiper, you write "they receive a Chunk which contains N records".

Is it possible to set a maximum value for N?

[edsiper]

it's not possible, our chunks are near 2MB, all records in the chunk are under the same tag

[LionelCons]

Then, is it possible to reduce the chunk size?

I found the Mem_Buf_Limit option but that seems to be at input level.

[LionelCons]

FWIW, I have made some tests with a big log file.

I saw an average size of 3.5 MB for the POST requests. This of course depends on formatting options (like Json_Date_Format).

This does not seem much but in an infrastructure with many fluent-bit instances sending their data in parallel this can be hard to swallow. We use Flume and the JSON input has to be parsed and enriched so it requires much more memory than its sheer size.

[LionelCons]

FWIW, this is very similar to #1938.

If we can control the max chunk size (per output plugin) then we can indirectly control the size of the JSON payload.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[LionelCons]

This issue is still creating problems in some of our workflows.

[teeohhem]

Running into this issue as well. Here's my vote

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[mrueg]

Not stale

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[mrueg]

Not stale