super_mediator
==============

super_mediator is an IPFIX mediator for use with YAF, SiLK, Analysis Pipeline,
and Mothra.  It processes YAF output data (by reading IPFIX files, by polling
a directory for IPFIX files, or by listening for TCP, UDP, or Spread from a
YAF process) and exports that data in IPFIX, JSON, or CSV Text format to one
or more collectors (e.g. flowcap, rwflowpack, Mothra, another super_mediator)
or to files (e.g. bulk upload to database).

super_mediator may be configured to pull the Deep Packet Inspection (DPI) data
from YAF that SiLK cannot collect and export that information to another IPFIX
collector or simply export the data to a CSV/JSON file for bulk upload into a
database of your choice.  Given MySQL credentials, super_mediator will import
the files into the given database.

super_mediator may also be configured to perform de-duplication on DPI
protocol information exported by YAF.  It will export the de-duplicated
records in IPFIX, CSV, or JSON format.  See the manual pages for more
information.

super_mediator provides simple command line arguments to do simple mediation
between one collector to one exporter.  To leverage the full power of
super_mediator and/or to use more than one collector and one exporter, use a
configuration file.  The file's syntax is described in the super_mediator.conf
manual page.

super_mediator can provide simple filtering on collection or at export time.
When built with SiLK support, this filtering includes the checking for IP
addresses in an IPSet.  See "SiLK Features" below.

To have super_mediator export data to a MySQL database, build super_mediator
with MySQL or MariaDB support.  See "Optional Libraries" under "Building"
below.

If compiled with OpenSSL support, super_mediator may be configured to compute
MD5 or SHA1 hashes of binary TLS/SSL X509 certificates when YAF is configured
to export the complete certificate.  See "Optional Libraries".

SiLK Features
-------------

The SiLK features of super_mediator are greatly increased in version 1.9.

Traditionally, super_mediator has been able to filter records based on whether
the record's IP addresses appear in a SiLK IPSet.  This feature required
access to either the complete SiLK installation or the standalone SiLK IPSet
library installation.

As of version 1.9, building with SiLK support gives super_mediator the
capability of modifying a record's deep packet inspection subTemplateMultiList
as follows:

* Adding sub-records that include a silkFlowtypeId and silkSensorId similar to
  how SiLK's rwflowpack tool labels records.  This is called Type and Sensor
  Labeling.  The algorithm used to label records is similar to that of the
  "twoway" packing logic.  For information on the SiLK tools, see
  https://tools.netsa.cert.org/silk/rwflowpack.html
  https://tools.netsa.cert.org/silk/packlogic-twoway.html

* Adding sub-records to note whether the record's IP addresses were present in
  an IPSet (IPSet Labeling) The name of the IPSet or an alternate label is
  included in the sub-record.

* Adding sub-records that record the Prefix Map (pmap) labels associated with
  either the IPFIX record's source and destination IP addresses or the
  record's protocolIdentifier paired with the sourceTransportPort and
  destinationTransportPort.  This is Prefix Map Labeling.  The sub-record
  includes the name of the Prefix Map file or an alternate label.

See the "SiLK Support" section below for details on including or excluding
this feature when building super_mediator.


Binary Distribution
===================

The CERT Linux Forensics Tools Repository (LiFTeR) provides binary
distributions for RPM-based systems (Fedora, RedHat, etc).  Follow the
instructions at https://forensics.cert.org/ to access these files.


Building
========

super_mediator uses a standard autotools-based build system.  The customary
build procedure (./configure && make && make install) should work in most
environments.

Required Libraries
------------------

These libraries must be built and installed before building super_mediator.

super_mediator requires GLib 2.18.0 or later.  GLib is included in many
operating environments and ports collections or it may be downloaded from
http://www.gtk.org/.  Ensure the development package is installed (e.g.,
glib2-devel or libglib2.0-dev) if your ports- or package-manager creates
separate packages for them.

super_mediator requires libfixbuf 2.3.0 or later.  libfixbuf may be downloaded
from https://tools.netsa.cert.org/fixbuf/.

When building, pkg-config is used to find libfixbuf.  You may need to set the
PKG_CONFIG_PATH to the directory containing the libfixbuf.pc file.

Optional Libraries
------------------

To allow reading of gzip-compressed IPFIX files, ensure the zlib library is
installed (and its development libraries and headers if needed).

Spread support requires Spread 4.1 or later.  Build and install the Spread
Toolkit before building libfixbuf and super_mediator if Spread is your desired
transport protocol.  Run ./configure --with-spread to enable Spread in the
super_mediator; libfixbuf must also be built with Spread support.  The
--with-spread switch accepts an optional argument specifying the installation
location of the toolkit, and the --with-spread-include and --with-spread-lib
options provide finer-grained control.  (Spread support has been removed from
super_mediator-2.0.0.)

If MySQL or MariaDB libraries are available, the super_table_creator program
will also be built.  Given a few MySQL parameters (name, password, database)
the super_table_creator will create a database and the necessary tables for
using the default super_mediator DPI CSV output.  To disable building the
super_table_creator, configure using --with-mysql=no.  You may supply the
complete path to the mysql_config utility as an argument to --with-openssl.

Building with OpenSSL support allows super_mediator to compute SHA1 and MD5
checksums of a binary X509 certificate.  Specify --with-openssl to enable this
feature.  You may supply the directory containing the OpenSSL installation as
an argument to --with-openssl.

SiLK Support
------------

To enable the capabilities described in "SiLK Features" above, super_mediator
must be built with either SiLK or the stand-alone SiLK IPSet support.

All features may be enabled when building with a complete SiLK installation.
When building with a SiLK IPSet installation, only IPSet Labeling and IPSet
filtering support is available.

The configure script always attempts to include the "SiLK Features" to the
greatest extend possible.  To disable the SiLK Features completely, specify
the --without-skipset switch to configure.

If you want to include this support but discover that the configure script is
unable to find the library you want it to use, specify --with-skipset=SILK_DIR
where SILK_DIR is the directory where either SiLK or SiLK IPSet is installed.
This switch may be included with any of the following options.

If you want to force super_mediator to use only the stand-alone SiLK IPSet
library, specify --without-libsilk.

When the SiLK library is used, the new labeling features in version 1.9 may be
independently disabled:

To disable IPSet filtering and IPSet labeling, specify --without-silk-ipset.

To disable Prefix Map labeling, specify --without-silk-pmap.

To disable Type and Sensor labeling, specify --without-silk-type-sensor.


Known Issues
=============

Please send bug reports, feature requests, and questions to
<[email protected]>.


Copyright and License
=====================

@DISTRIBUTION_STATEMENT_BEGIN@
super_mediator-1.11

Copyright 2024 Carnegie Mellon University.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING
INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON
UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED,
AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR
PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF
THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF
ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT
INFRINGEMENT.

Licensed under a GNU GPL 2.0-style license, please see LICENSE.txt or
contact [email protected] for full terms.

[DISTRIBUTION STATEMENT A] This material has been approved for public
release and unlimited distribution.  Please see Copyright notice for
non-US Government use and distribution.

This Software includes and/or makes use of Third-Party Software each
subject to its own license.

DM24-1038
@DISTRIBUTION_STATEMENT_END@