Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter_nest: add wildcard_exclude config #10112

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

0x006EA1E5
Copy link
Contributor

@0x006EA1E5 0x006EA1E5 commented Mar 20, 2025

Implements #10111 by adding wildcard_exclude config to nest filter

#10111

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@0x006EA1E5
Copy link
Contributor Author

0x006EA1E5 commented Mar 20, 2025

Example config:

pipeline:
  inputs:
    - name: dummy
      tag: logs
      dummy: |
        {
          "message": "simple log generated",
          "logger": "my.logger",
          "level": "INFO",
          "hostname": "localhost",
          "my_map_of_attributes_1": {
            "key_1": "hello, world!",
            "key_2": "goodbye, world!"
          },
          "my_map_of_maps_1": {
            "root_key": {
              "sub_key_1": "hello, world!",
              "sub_key_2": "goodbye, world!"
            }
          },
          "an_unknown_key": "hello, world!",
          "another_unknown_key": "goodbye, world!"
        }

  filters:
    - name: nest
      match: logs
      nest_under: nested
      wildcard: '*'
      wildcard_exclude:
        - message
        - logger
        - hostname
        - level
        - 'my_map_of_*'

  outputs:
    - name: stdout
      match: logs

The above will nest keys an_unknown_key and another_unknown_key under nested

@0x006EA1E5
Copy link
Contributor Author

I have added this config to the docker compose for structured_metadata.

Here is a screenshot showing that the end result is as expected:
image

(an_unknown_key and another_unknown_key are ultimately used as structured_metadata)

@0x006EA1E5
Copy link
Contributor Author

0x006EA1E5 commented Mar 20, 2025

Here is some console output, showing how the above example config produces the correct output:

fluentbit  | [0] logs: [[1742483011.135625671, {}], {"message"=>"simple log generated", "logger"=>"my.logger", "level"=>"INFO", "hostname"=>"localhost", "my_map_of_attributes_1"=>{"key_1"=>"hello, world!", "key_2"=>"goodbye, world!"}, "my_map_of_maps_1"=>{"root_key"=>{"sub_key_1"=>"hello, world!", "sub_key_2"=>"goodbye, world!"}}, "nested"=>{"an_unknown_key"=>"hello, world!", "another_unknown_key"=>"goodbye, world!"}}]


if (wildcard_exclude->key_is_dynamic) {
/* This will negatively match "ABC123" with prefix "ABC*" */
if (strncmp(key, wildcard_exclude->key, wildcard_exclude->key_len) == 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key is not a null terminated string and this comparison can potentially fail. Since klen is set, I recommend comparing first klen == wildcard_exclude->key_len and if it passes then do the strncmp()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants