Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter_logs_to_metrics: document field_value usage for counters #1584

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

filter_logs_to_metrics: document field_value usage for counters #1584

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dfaf16d
filter_logs_to_metrics: document field_value usage for counters
databus23 Mar 10, 2025
13f50c0
Apply suggestions from code review
databus23 Mar 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions pipeline/filters/log_to_metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ description: Generate metrics from logs

## Log To Metrics

The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to count records, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline.
The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to create counters based on the number of records or the value of a field within those records, create a gauge for field values, or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a separate metric pipeline.

_Please note that this plugin is an experimental feature and is not recommended for production use. Configuration parameters and plugin functionality are subject to change without notice._

Expand All @@ -25,7 +25,7 @@ The plugin supports the following configuration parameters:
| bucket | Defines a bucket for `histogram` | Yes, for mode `histogram` | e.g. 0.75 |
| add\_label | Add a custom label NAME and set the value to the value of KEY | | |
| label\_field | Includes a record field as label dimension in the metric. | | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
| value\_field | Specify the record field that holds a numerical value | Yes, for modes \[`gauge` and `histogram`] | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
| value\_field | Specify the record field that holds a numerical value | Yes, for modes `gauge` and `histogram`, optional for `counter` | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
| kubernetes\_mode | If enabled, it will automatically put pod\_id, pod\_name, namespace\_name, docker\_id and container\_name into the metric as labels. This option is intended to be used in combination with the [kubernetes](kubernetes.md) filter plugin, which fills those fields. | | |
| Regex | Include records in which the content of KEY matches the regular expression. | | KEY REGEX |
| Exclude | Exclude records in which the content of KEY matches the regular expression. | | KEY REGEX |
Expand All @@ -34,7 +34,7 @@ The plugin supports the following configuration parameters:

### Getting Started

The following example takes records from two dummy inputs and counts all messages passing through the `log_to_metrics` filter. It then generates metric records which are provided to the `prometheus_exporter`:
The following example takes records from two dummy inputs and generates two different counter metrics: the number of messages that pass through the `log_to_metrics` filter, and the combined value of the `duration` fields included in each of those messages. It then generates metric records which are provided to the `prometheus_exporter`:

#### Configuration - Counter

Expand All @@ -61,6 +61,15 @@ The following example takes records from two dummy inputs and counts all message
metric_name count_all_dummy_messages
metric_description This metric counts dummy messages

[FILTER]
name log_to_metrics
match dummy.log*
tag test_metric
metric_mode counter
metric_name total_duration
metric_description This metric counts the total time spent across all messages
value_field duration

[OUTPUT]
name prometheus_exporter
match *
Expand All @@ -76,7 +85,10 @@ You can then use e.g. curl command to retrieve the generated metric:

# HELP log_metric_counter_count_all_dummy_messages This metric counts dummy messages
# TYPE log_metric_counter_count_all_dummy_messages counter
log_metric_counter_count_all_dummy_messages 49
log_metric_counter_count_all_dummy_messages 24
# HELP log_metric_counter_total_duration This metric counts the total time spend
# TYPE log_metric_counter_total_duration counter
log_metric_counter_total_duration 960
```

#### Configuration - Gauge
Expand Down