Using dedupe inside a batch in an S3 Output?

[pritster5]

Hi all, is it possible to dedupe batched messages conditionally inside an s3 output? I'm trying to do something like this:

output_resources:
  - label: s3_output
    aws_s3:
      bucket: ${AWS_BUCKET}
      path: ${AWS_BUCKET_PATH}/blah-${! meta("blah") }-${! meta("blahblah") }-${! nanoid(5) }.txt
      region: ${AWS_REGION}
      credentials:
        id: ${AWS_ACCESS_ID}
        secret: ${AWS_SECRET_ID}
      batching:
        period: 5s
        byte_size: 4194304 # 64MB
        processors:
          - switch:
            - check: env("SPLUNK_SOURCETYPE") == "aws:cloudwatchlogs:vpcflow"
              processors:
                - dedupe: 
                    cache: insert_id_cache
                    key: blah-${! json("insertId") }

but I'm not sure if the switch/check clause prevents any messages that do NOT satisfy the condition from processing. Do I need to add some kind of default case after the check?

[mihaitodor]

Hey @pritster5 👋

but I'm not sure if the switch/check clause prevents any messages that do NOT satisfy the condition from processing. Do I need to add some kind of default case after the check?

Those remain unchanged. The processors under the switch case you have there only act on the messages which match the check.