New Module: PCAP capture

[romaindoumenc]

Some traffic flows should always be captured (e.g. for auditing purposes).
The built-in captures modules (logger and network_capture) seems a bit too coarse for our use case.

Let’s add a new module to capture pcap traces of fine-grained filters (at least source IP, destination IP and port).

Implementation

• The module maintains a list of PCAP filters of interest.

• For each new packet, the eval method would gate the packet to prevent too many unnecessary hits (for now, using the destination IP), then iterate over all filters of interest. When a filter matches, the packet is added to the trace.

• The files should be rotated at a size chosen in the configuration.

• The filters can be configured in the Lua file, and dynamically (via the command probably).

[mike-at-trout-software]

A couple of questions:

1. Can we just look at the first package in a flow, or would there be cases where we want to pcap "random" packages from a flow - if we can just look at the first package it would significantly reduce the processing power needed to do the filtering

2. Do we want to handle the case where a given package/flow matches multiple filters, i.e. should be written to multiple pcaps?

[romaindoumenc]

Good questions!

1. The kind of filters we want to perform (TCP port / IP address filtering) cannot change during the lifetime of the session – this is not an arbitrary filter system. So yes, it is perfectly fine to check initially, but then, if the flow match, further packets should be added to the same trace. I don’t know if you had in mind a specific kind of snort module – I seem to recall some only run on packet start, or just a flow-level state variable to notice if tracked or not. Either way, determine early, keep then.
2. The “front-end” (technically SH back-end, but your front-end is my back-end) will ensure that there is only one expression matching a given filter, so we should not worry about that.

[mike-at-trout-software]

1. The kind of filters we want to perform (TCP port / IP address filtering) cannot change during the lifetime of the session – this is not an arbitrary filter system. So yes, it is perfectly fine to check initially, but then, if the flow match, further packets should be added to the same trace. I don’t know if you had in mind a specific kind of snort module – I seem to recall some only run on packet start, or just a flow-level state variable to notice if tracked or not. Either way, determine early, keep then.

Yes, there is a configuration where a plugin only sees the first packet, I did not plan to use that feature - I was more thinking along the lines of using flow data, so when I get a packet, I check if there is a decision to log it (and where to log it) in the flow data. So the algorithm would go something like:

A1) Plugin receives package from snort
A1) Attempt to read decision from flow
A2) If (no decision found in flow data):
A3)     Run package through filtering algorithm
A4)     Persist result of filtering algorithm in flow data (store decision)
A5) If decision is to not log flow, skip package and return
A6) Write package to pcap as specified in stored decision and return

2. The “front-end” (technically SH back-end, but your front-end is my back-end) will ensure that there is only one expression matching a given filter, so we should not worry about that.

Ok, then I'll stop the filter algorithm at the first match, and only make room for one match in the flow data

[mike-at-trout-software]

For the dynamic update we are (at least initially) aiming to do it by updating the lua config(s) and then sending a reload (1) event to snort to read the updated lua configs.

[mike-at-trout-software]

@romaindoumenc for the rotating filenames, would something like a <base name from config><time><date>.pcap format be sufficient, so we avoid overwriting stuff, if the config is reloaded due to changes that might be unrelated to the pcap capture configuration, or if snort is just restarted for some reason.

Also would it be ok, to put the size limit, before a new file is generate, on the sum of the length of the packages written, rather than the actual file size?

[mike-at-trout-software]

@romaindoumenc when we reload the configuration (lua scripts) should that impact existing streams? - I'm thinking about the case where a long lived stream is active when the configuration is changed, should it be possible to enable/disable logging of this long lived stream?

I see two main issues if we don't allow the configuration changes to be applied to the existing streams:

1. A stream generating a lot of data can't be stopped from writing a lot of data
2. Streams already established at the time of the configuration change, will never get logged, even they are in the current filter.

[romaindoumenc]

Good thinking @mike-at-trout-software .

I understand your concerns, but I think the use case to enable changes in an existing stream is exceedingly rare from the Lua side (which requires manual user intervention).
I do think we will revisit this when we do automated flow detection (think “agent decides this flow is suspect and starts recording it”).

[mike-at-trout-software]

For reference, sample config.lua for the module:

    stream = {}
    stream_icmp = {}
    stream_tcp = {}
    stream_udp = {}
    stream_ip = {}
    
    capture_pcap = {
      snap_length = 4096,
      optimize_filter = true,
      rotate_limit = 1000000,
      map = {
        { filter = "net 161.35.18.220",
          --hint_ip = "161.35.18.220",
          --hint_port = "80",
          pcap_prefix = "Prefix_220_"
        }, {
          filter = "net 1.1.1.1",
          pcap_prefix = "Prefix_1_"
        }
      }
    }