Using kubernetes filter and cleaning log entry

[idomic]

Bug Report

I tried adding the kubernetes filter so I can have the pod and container name as keys for using log_stream_template $kubernetes['pod_name'].$kubernetes['container_name'].
I then tried using the log_key log so I can get only the actual logs in cloud watch, following the docs.
It seems to break the log_stream_template since the keys are not there any more. I wanted a way to have only the log entries saved and not the whole kubernetes json, with a custom log stream name, based on the pod and container names.

Describe the bug
Mentioned above

To Reproduce
Use both the log_key and the log_stream_template in the cloudwatch output.

• Example log message if applicable:

{
    "time": "2025-01-13T20:08:58.990820577Z",
    "stream": "stdout",
    "_p": "F",
    "log": "WARNING: Static file serving is enabled, but no static folder found at /srv/static. To disable static file serving, set server.enableStaticServing to false.",
    "kubernetes": {
        "pod_name": "ancient-forest-6576-6b446c9b69-k7t4h",
        "namespace_name": "apps",
        "pod_id": "73213877-33da-485f-98a4-accb24b938ec",
        "labels": {
            "app": "app",
            "pod-template-hash": "6b446c9b69",
            "version": "ancient-forest-6576"
        },
        "host": "ip-10-0-126-130.ec2.internal",
        "container_name": "app-container",
        "docker_id": "c0306b4ceb53949cd11a0423d99919dcde878f41213e66ba4bb44d86993fe82e",
        "container_hash": "account.dkr.ecr.us-east-1.amazonaws.com/ancient-forest-6576@sha256:c4614a87553991bdb8cef08c2194b504a042b60360fec8e0be364fd66fdecb4c",
        "container_image": "account.dkr.ecr.us-east-1.amazonaws.com/ancient-forest-6576:f05f087d"
    }
}

• Steps to reproduce the problem:

Expected behavior
Being able to set the custom name and remove all of the keys from the log.

Screenshots
NA

Your Environment

• Version used: EKS, (1.9, public.ecr.aws/aws-observability/aws-for-fluent-bit:2.32.4 )
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): Kuberenetes
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[idomic]

That's the conf I'm using:

  fluent-bit.conf: |
    [SERVICE]
        Flush                     5
        Grace                     30
        Log_Level                 debug
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               ${HTTP_SERVER}
        HTTP_Listen               0.0.0.0
        HTTP_Port                 ${HTTP_PORT}
        storage.path              /var/fluent-bit/state/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 5M

    @INCLUDE application-log.conf
    @INCLUDE dataplane-log.conf
    @INCLUDE host-log.conf

  application-log.conf: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log
        multiline.parser    docker, cri
        DB                  /var/fluent-bit/state/flb_container.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      ${READ_FROM_HEAD}
    [INPUT]
        Name                tail
        Tag                 kube.*
        Path                /var/log/containers/cloudwatch-agent*
        multiline.parser    docker, cri
        DB                  /var/fluent-bit/state/flb_cwagent.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}
    
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Keep_Log            Off
        Buffer_Size         64KB
    
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Keep_Log            Off
        Buffer_Size         64KB
    
    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /eks/app-logs
        log_stream_prefix   fallback-
        auto_create_group   On
        log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
        extra_user_agent    container-insights
    
    [OUTPUT]
        Name                cloudwatch_logs
        Match               kube.*
        region              ${AWS_REGION}
        log_group_name      /eks/app-logs
        log_stream_prefix   fallback-
        auto_create_group   On
        log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
        extra_user_agent    container-insights

  dataplane-log.conf: |
    [INPUT]
        Name                systemd
        Tag                 dataplane.systemd.*
        Systemd_Filter      _SYSTEMD_UNIT=docker.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        DB                  /var/fluent-bit/state/systemd.db
        Path                /var/log/journal
        Read_From_Tail      ${READ_FROM_TAIL}

    [INPUT]
        Name                tail
        Tag                 dataplane.tail.*
        Path                /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        multiline.parser    docker, cri
        DB                  /var/fluent-bit/state/flb_dataplane_tail.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      ${READ_FROM_HEAD}

    [FILTER]
        Name                modify
        Match               dataplane.systemd.*
        Rename              _HOSTNAME                   hostname
        Rename              _SYSTEMD_UNIT               systemd_unit
        Rename              MESSAGE                     message
        Remove_regex        ^((?!hostname|systemd_unit|message).)*$

    [FILTER]
        Name                aws
        Match               dataplane.*
        imds_version        v2

    [OUTPUT]
        Name                cloudwatch_logs
        Match               dataplane.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/dataplane
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights

  host-log.conf: |
    [INPUT]
        Name                tail
        Tag                 host.dmesg
        Path                /var/log/dmesg
        Key                 message
        DB                  /var/fluent-bit/state/flb_dmesg.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}

    [INPUT]
        Name                tail
        Tag                 host.messages
        Path                /var/log/messages
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_messages.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}

    [INPUT]
        Name                tail
        Tag                 host.secure
        Path                /var/log/secure
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_secure.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}

    [FILTER]
        Name                aws
        Match               host.*
        imds_version        v2

    [OUTPUT]
        Name                cloudwatch_logs
        Match               host.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/host
        log_stream_prefix   ${HOST_NAME}.
        auto_create_group   true
        extra_user_agent    container-insights

  parsers.conf: |
    [PARSER]
        Name                syslog
        Format              regex
        Regex               ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key            time
        Time_Format         %b %d %H:%M:%S

    [PARSER]
        Name                container_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

    [PARSER]
        Name                cwagent_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ
    
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

[patrick-stephens]

Can you repeat with the OSS image latest version (3.2.4) and also using log_key log in the output?

[idomic]

It worked once I've switched to the latest (3.2.4) and adding the log_key (I double checked and it doesn't work with the AWS image). Closing this.