How to extract TAG components and add them as attributes to the record?

[sfc-gh-aivanou]

Hello! I searched in github and surprisingly I did not find this pattern, that should be pretty common.

I have a tag that has the following pattern:

my_prefix.<PROCESS_NAME>

How do I add <PROCESS_NAME> as attribute to the record?

E.g. I want to produce the following record:

my_prefix.<PROCESS_NAME>.data.log: [[1712008024.991679581, {}], {"message"=>"my-message", "process_name": "<PROCESS_NAME>"} 

[Rodeoclash]

I also seem to be having a bit of trouble finding where in the documentation I should be looking for this. In my case, I'm using the fluentd logging engine in Docker to ship my logs to a fluent-bit instance:

Terraform config

  log_configuration = {
    logDriver = "fluentd"
    options = {
      "fluentd-address" = "localhost:24224",
      "tag"             = "xonboard_backend2"
    }
    secretOptions = null
  }

I then want to extract the tag that's been passed and set it as the service.name field:

[FILTER]
    Name   modify
    Match  *
    Set    service.name test

In fluentd this was as possible using ${tag} but I'm having a bit more trouble in fluent-bit land. @sfc-gh-aivanou did you find a solution to this? I feel like we're in a similar spot 🙂

[sfc-gh-aivanou]

hi.
I used lua script to achieve my goal:

  filters.conf: |
    [FILTER]
        Name lua
        Match *
        script filters.lua
        call  my_script

  filters.lua: |
    function my_script(tag, timestamp, record)
        local parts = {}
        for part in tag:gmatch("[^.]+") do
            table.insert(parts, part)
        end
        record.my_tag = parts[0]
        return 2, timestamp, record
    end

[sparrowt]

For anyone coming here wishing to simply extract the full tag and add it as a field to the record, the example here in the docs seems to do this - reproducing the yaml config here in case the link breaks in future:

pipeline:
    inputs:
        ...

    filters:
        - name:  lua
          match: "*"
          call:  append_tag
          code:  |
              function append_tag(tag, timestamp, record)
                 new_record = record
                 new_record["tag"] = tag
                 return 1, timestamp, new_record
              end

This allows you to then reference tag as a normal field on the record, for example:

    outputs:
        - name: opensearch
          logstash_format: on
          logstash_prefix_key: tag
          logstash_dateformat: 'mycustomsuffix'  # deliberately don't include any strfime format symbols to exclude the date

which results in dynamic index names based on the tag, e.g. the above produces: thetagvalue-mycustomsuffix

[sparrowt]

Since writing this however, I have discovered that there is a magic $TAG which works out of the box, so if you want a custom elastic/opensearch index name based on the tag then you can use that much more easily as described here.