GSP
Quick Navigator

Search Site

Unix VPS
A - Starter
B - Basic
C - Preferred
D - Commercial
MPS - Dedicated
Previous VPSs
* Sign Up! *

Support
Contact Us
Online Help
Handbooks
Domain Status
Man Pages

FAQ
Virtual Servers
Pricing
Billing
Technical

Network
Facilities
Connectivity
Topology Map

Miscellaneous
Server Agreement
Year 2038
Credits
 

USA Flag

 

 

Man Pages
SUPER_MEDIATOR(1) ipfix mediator SUPER_MEDIATOR(1)

super_mediator - IPFIX Mediator

     super_mediator   [--config CONFIG_FILE_NAME]
                      [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER]
                      [--ipfix-port PORT] [--export-port PORT]
                      [--ipfix-input TRANSPORT_PROTOCOL]
                      [--output-mode TRANSPORT_PROTOCOL]
                      [--watch POLL_TIME]
                      [--move PROCESSED_INPUT_DIRECTORY] [--lock]
                      [--rotate ROTATE_SECONDS]
                      [--no-stats] [--dns-dedup]
                      [--groups SPREAD_GROUPS]
                      [--udp-temp-timeout TIMEOUT_SECONDS]
                      [--log LOG_SPECIFIER]
                      [--log-dir LOG_PATH]
                      [--daemonize] [--pidfile PIDFILE_NAME]
                      [--fields FIELD_LIST]
                      [--print-headers]
                      [--sleep MICROSECONDS]
                      [--verbose] [--quiet] [--version]
                      [--become-user UNPRIVILEGED_USER]                                   
                      [--become-group UNPRIVILEGED_GROUP]  
                      [--metadata-export]

super_mediator is an IPFIX Mediator to be used as a manifold for yaf(1) IPFIX export data. It has multiple means of reading and writing IPFIX. A limited number of options can be specified on the command line. For advanced configuration with multiple collectors, multiple exporters, and filtering, the configuration file super_mediator.conf(1) should be used. Command line arguments for collection will override collectors defined in the configuration file. Command line arguments for exporters will be ignored if a configuration file is provided.

super_mediator can read from an IPFIX file, watch for files in a given directory, collect IPFIX from yaf(1) via TCP or UDP on a given host and port, or subscribe to a series of Spread Groups. It can have multiple incoming sources and export to multiple collectors. It can export to a single or rotating IPFIX, CSV, or JSON files, via UDP or TCP, and to multiple Spread Groups.

super_mediator can filter on import and/or export. Filters can be defined in the configuration file.

super_mediator is capable of collecting all of YAF Deep Packet Inspection data (see yafdpi(1)). For advanced configuration of the DPI to CSV output, see super_mediator.conf(1). For DNS, the super_mediator can perform de-duplication on the incoming DNS resource records. It will cache a key containing the resource record name (rrname), resource record type (rrtype), and the resource record value (rrval). When a resource record is seen matching this key, the super_mediator will increment the hit count and the last_seen timestamp associated with the record. If it is a new record, the record will be exported. Each record will be exported to the appropriate exporter. The text format is:

    first_seen | rrtype | rrname | rrval

(see below for IPFIX template)

Records will be periodically flushed. The default is to flush a record from the buffer once it has reached the desired hit count of 500 or a new record has not been seen in 5 minutes. These default behaviors can be modified in the configuration file. Similarly, a file can be written upon flush time. If this is the case, the text format would be:

    first_seen | last_seen | rrtype | rrname | hitcount | rrval

There is also the option to BASE 64 Encode all DNS domain names in a text file. This option can be specified in the configuration file.

When possible, the configuration file should be used when running the super_mediator. However, some command line options were given as a convenience for quick analysis.
--config CONFIGURATION_FILE
CONFIGURATION_FILE is the configuration file following the guidelines given in the super_mediator.conf man page. If this option is given, it should be the only command line option given.

These options control where super_mediator will take its input from. super_mediator can read packets from a file, directory, live on a TCP or UDP port, or by subscribing to Spread group(s) through the Spread daemon. By default, if no input options are given, super_mediator reads an IPFIX file on standard input.
--in INPUT_SPECIFIER
INPUT_SPECIFIER is an input specifier. If --ipfix-input is given and set to TCP or UDP, INPUT_SPECIFIER is the hostname or IP Address of the host to listen on. If --ipfix-input is SPREAD, INPUT_SPECIFIER is the daemon name of the Spread daemon to connect to. If the Spread daemon is running on remote host, the remote host or IP should be given in the form daemon_name@hostname. If --watch is given, INPUT_SPECIFIER should be a file glob pattern, which must be escaped or quoted to prevent shell expansion. Files that match this pattern will be processed by the super_mediator. If no other options are given, the super_mediator assumes INPUT_SPECIFIER is an IPFIX file.
--ipfix-port PORT
If <--ipfix-input> is present, export flows to TCP or UDP on port port. If not present, the default port 18000 is used.
--ipfix-input TRANSPORT_PROTOCOL
If present, causes super_mediator to operate as an IPFIX collector, listening for connections via the sepecified protocol TRANSPORT_PROTOCOL from a yaf(1) exporter named in the INPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp, udp, and spread; spread is only available if super_mediator was built with Spread support. UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages. If spread is specified, --groups must also be present.
--watch POLL_TIME
If present, process files that match the pattern given to --in every POLL_TIME seconds. super_mediator will run forever waiting for files that match the pattern. --watch should be used with --move. If --move is not present, files will be deleted after they have been processed.
--move PROCESSED_INPUT_DIRECTORY
If present, input files will be moved to PROCESSED_INPUT_DIRECTORY after they have been successfully processed. If this is not present with --watch, files will be deleted after they are processed.
--lock
If present, super_mediator will not read files that are locked, which means they have the extension ".lock" appended to the end of the filename. This can be used if super_mediator is reading from a yaf(1) export directory and yaf(1) is run with --lock. This will prevent super_mediator from removing the files out from under yaf(1). This does not lock files that the super_mediator is writing to. Use the super_mediator configuration file to enable locking of output files.
--groups SPREAD_GROUPS
If --ipfix-input is present and set to spread, use --group to specify the the Spread group name(s) to subscribe to. It is possible to list more than one group name in a comma-seperated list. See the Spread Documentation, www.spread.org, for more details on Spread.

These options control where super_mediator will send its output. super_mediator can write flows to an IPFIX file, text file, or to an IPFIX collector over TCP, UDP, or Spread. By default, if no options are given, yaf(1) writes IPFIX to standard out.
--out OUTPUT_SPECIFIER
OUTPUT_SPECIFIER is an output specifier. If --output-mode is present, and set to TCP or UDP, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. Otherwise, if --output-mode is set to SPREAD, the OUTPUT_SPECIFIER should be the Spread daemon name. If the Spread daemon name is running on a remote host, it should be in the form daemon_name@hostname. If --output-mode is set to TEXT, OUTPUT_SPECIFIER is a filename in which the flows will be written in pipe-delimited format. Otherwise, OUTPUT_SPECIFIER is a filename in which flows will be written in IPFIX Format. The string - may be used to write to standard output (the default). If <--rotate> is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. super_mediator must be able to make an initial connection to the OUTPUT_SPECIFIER for super_mediator to start. If the connection is lost after the initial connection, super_mediator will immediately retry the connection after reporting a warning message to the log. If the retry is unsuccessful, super_mediator will retry the connection every 15 seconds until the connection is successful. Flows will be lost while the connection is down.
--export-port PORT
If --output-mode is present and set to TCP or UDP, export flows to port PORT. If not present, the default port 18001 will be used.
--output-mode TRANSPORT_PROTOCOL
If present, causes super_mediator to operate as an IPFIX or TEXT Exporter, exporting via the specified protocol TRANSPORT_PROTOCOL to a collector (e.g rwflowpack, flowcap) named in the OUTPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp, udp, text, and json; spread is only available through the configuration file. UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages.
--udp-temp-timeout TIMEOUT_SECS
Set UDP template timeout in seconds if --ipfix is set to udp. As per RFC 5101 recommendations, super_mediator will attempt to export templates three times within TEMPLATE_TIMEOUT. The default template timeout period is 600 seconds (10 minutes).
--no-stats
If present, super_mediator will not forward yaf(1) process statistics records or log statistics. It is possible to configure certain exporters to process stats while others ignore stats messages. This must be done with through the super_mediator configuration file.
--sleep MICROSECONDS
If present, super_mediator will sleep for MICROSECONDS between each call to fBufAppend, which appends the IPFIX messages to the output source. This is useful if super_mediator is reading an IPFIX file and transmitting IPFIX over UDP. super_mediator may send the messages too quickly for the IPFIX Collector to receive them (possibly dropping messages.) This option is only available with one collector and one exporter when executing super_mediator from the command line.
--fields FIELD_LIST
If present and --output-mode=TEXT is also present, write only the fields given in FIELD_LIST. FIELD_LIST is a list of integers corresponding to flow fields, separated by a comma. The list of acceptable fields are listed in super_mediator.conf(1) under FIELDS. Only the integer representation of the field is accepted. The below example will print: stime|etime|sip|dip|sport|dport|protocol|applabel for each flow to the file given to --out.
--fields 18,19,0,1,4,5,6,7
--metadata-export
If present, super_mediator will include information element and template metadata in output. It is possible to configure certain exporters to include this metadata while others do not via the super_mediator configuration file.
--print-headers
If present for TEXT Exporters, the super_mediator will write a header for delimited flow data. If files rotate, it will write one header at the top of each flow data file. Ignored for custom field lists.

--log LOG_SPECIFIER
Specifies the destination for log messages. LOG_FILE can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr. The log level can be specified by the LOGLEVEL keyword in the Super_mediator configuration file or by using one of the following two options. Default level is WARNING.
--log-dir LOG_PATH
If present, super_mediator will write log files to LOG_PATH. LOG_PATH must be a complete directory path. The log files have the form

LOG_PATH/sm-YYYYMMDD.log

where YYYYMMDD is the current date. The log files are rotated at midnight local time. When the log files are rotated a new log is opened, the previous file is closed, and gzip(1) is invoked on the previous day's log file. (Old log files will not be removed by super_mediator.)

--verbose
If present, log all messages. The default log level is WARNING. This option will change the log level to DEBUG and log all yaf(1) and super_mediator process statistics, along with any IO operations.
--quiet
If present, turn off logging completely. super_mediator will not log errors.
--daemonize
If present, super_mediator will become a daemon.
--pidfile PIDFILE_NAME
Set the complete path to the file in which super_mediator writes its process ID (pid) when running as a daemon. --pid-file is ignored if --daemon is not present.

These options are used to cause super_mediator to drop privileges when running as root for live capture purposes.
--become-user UNPRIVILEGED_USER
After opening the live capture device in --live mode, drop privilege to the named user. Using --become-user requires super_mediator to be run as root or setuid root. This option will cause all files written by super_mediator to be owned by the user UNPRIVILEGED_USER and the user's primary group; use --become-group as well to change the group super_mediator runs as for output purposes.

If running as root for live capture purposes and --become-user is not present, super_mediator will warn that privilege is not being dropped. We highly recommend the use of this option, especially in production environments, for security purposes.

--become-group UNPRIVILEGED_GROUP
--become-group can be used to change the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.

--dns-dedup
If preset, super_mediator will cache DNS resource records and emit records only when new ones are seen or super_mediator has seen 500 of the same records, by default. DNS de-duplication can be further configured in the super_mediator configuration file.

The following IPFIX fields are exported by the super_mediator. DPI information will be exported in the same format as received by yaf(1), with the exception of the de-duplicated DNS records. Any statistics option messages will also be formatted in the same format as they are received.

FLOW_ONLY Record

Some fields are only exported if they are non-zero and were enabled in yaf. super_mediator exports many variations of the following template. The following list contains all of the possible elements that could exist in the FLOW_ONLY record. Reverse elements are only exported if reversePacketTotalCount or reversePacketDeltaCount is nonzero.

    flowStartMilliseconds, IE 152, 8 octets, unsigned

    flowEndMilliseconds, IE 153, 8 octets, unsigned

    octetTotalCount, IE 85, 8 octets, unsigned

    reverseOctetTotalCount Reverse IE 85 (PEN 29305), 8 octets, unsigned

    packetTotalcount, IE 86, 8 octets, unsigned

    reversePacketTotalcount, Reverse IE 86 (PEN 29305), 8 octets, unsigned

    octetDeltaCount, IE 1, 8 octets, unsigned

    reverseOctetDeltaCount, reverse IE 1 (PEN 29305), 8 octets, unsigned

    packetDeltaCount, IE 2, 8 octets, unsigned

    reversePacketDeltaCount, reverse IE 2 (PEN 29305), 8 octets, unsigned

    sourceIPv6Address, IE 27, 16 octets, unsigned

    destinationIPv6Address, IE 28, 16 octets, unsigned

    sourceIPv4Address, IE 8, 4 octets, unsigned

    destinationIPv4Address, IE 12, 4 octets, unsigned

    sourceTransportPort, IE 7, 2 octets, unsigned

    destinationTransportPort, IE 11, 2 octets, unsigned

    flowAttributes, CERT (PEN 6871) IE 40, 2 octets, unsigned

    reverseFlowAttributes, CERT (PEN 6871) IE 16424, 2 octets, unsigned

    protocolIdentifier, IE 4, 1 octet, unsigned

    flowEndReason, IE 136, 1 octet, unsigned

    silkAppLabel, CERT (PEN 6871) IE 33, 2 octets, unsigned

    reverseFlowDeltaMilliseconds, CERT (PEN 6871) IE 21, 4 octets, unsigned

    tcpSequenceNumber, IE 184, 4 octets, unsigned

    reverseTcpSequenceNumber, Reverse IE 184 (PEN 29305), 4 octets, unsigned

    initialTCPFlags, CERT (PEN 6871) IE 14, 1 octet, unsigned

    unionTCPFlags, CERT (PEN 6871) IE 15, 1 octet, unsigned

    reverseInitialTCPFlags, CERT (PEN 6871) IE 16398, 1 octet, unsigned

    reverseUnionTCPFlags, CERT (PEN 6871) IE 16399, 1 octet, unsigned

    vlanId, IE 58, 2 octets, unsigned

    reverseVlanId, Reverse IE 58 (PEN 29305), 2 octets, unsigned

    ipClassOfService, IE 5, 1 octet, unsigned

    reverseIpClassOfService, Reverse IE 15 (PEN 29305), 1 octet, unsigned

    mplsTopLabelStackSection, IE 70, 3 octets, unsigned

    mplsLabelStackSection2, IE 71, 3 octets, unsigned

    mplsLabelStackSection3, IE 72, 3 octets, unsigned

    observationDomainId, IE 149, 4 octets, unsigned

        The observation domain ID of the yaf(1) process that
        generated the flow.

    subTemplateMultiList, IE 293, Variable Length

DNS De-duplicated Record

    flowStartMilliseconds, IE 152, 8 octets, unsigned

        The time in milliseconds of when the DNS resource record was
        first seen.

    flowEndMilliseconds, IE 153, 8 octets, unsigned

        The time in milliseconds of the last record seen by the
        B<super_mediator>. This is only exported if the
        B<super_mediator> is configured to LAST_SEEN.

    sourceIPv4Address, IE 8, 4 octets, unsigned

        The IPv4 Address found in a DNS A Record.

    dnsTTL, CERT (PEN 6871) IE 199, 4 octets, unsigned

        The maximum TTL seen for the aggregated DNS records.
        This is only exported if the B<super_mediator> is
        configured to LAST_SEEN.

    dnsQRType, CERT (PEN 6871) IE 175, 2 octets, unsigned

        The type of Resource Record.  This corresponds
        with the QTYPE filed in the DNS Question Section
        or the TYP field in the DNS Resource Record Section.

    dnsHitCount, CERT (PEN 6871) IE 228, 2 octets, unsigned

        The number of times the B<super_mediator> saw
        this record in the FLUSH TIMEOUT period.
        This is only exported if the B<super_mediator> is
        configured to LAST_SEEN.

    dnsQName, CERT (PEN 6871) IE 179, variable length

        A DNS Response Name.  This field corresponds
        with the NAME field in the DNS Resource Record Section.

    dnsRName, CERT (PEN 6871) IE 927, variable length

       A DNS Resource Record Data Element.  This field
       corresponds with the RDATA in the DNS Resource
       Record Section.  For CNAME records, this will be the
       canonical name. For NS Records, this will be the
       name server name.  For AAAA records, this will be an
       IPv6 Address, etc.

    observationDomainName IE 300, variable length

       This field is populated if the DEDUP_CONFIG block was 
       configured with B<ADD_EXPORTER_NAME>
       or DNS_DEDUP was configured with a B<VLAN_MAP> or B<OBID_MAP>.
       See super_mediator.conf(1) for more information on MAPS.

Tombstone Options Template

Super_mediator will forward and can optionally generate tombstone records using IPFIX Options Template Records. These records are intended to allow the analysis of the time it takes for records to be processed by each tool (eg. YAF, Super Mediator, SiLK) in your environment. Each tombstone record generated by super_mediator consists of four information elements: a user-settable ID, an ID that is randomly generated for each run of super_mediator, a sequentially increasing ID for each record, and a subTemplateList of the time each program interacted with the tombstone record. With ideal randomness, the 3 IDs taken together should uniquely specify a record. Tombstone record generation is not on by default and can be turned on using the TOMBSTONE_CONFIG section of the config file. Tombstone Records are only sent/forwarded to exporters configured to receive stats records.

The following Information Elements will be exported:

    tombstoneId, CERT (PEN 6871) IE 550, 4 octets, unsigned

       A sequentially increasing identification number unique to each tombstone 
       record in a run of a given program.

    exporterConfiguredId, CERT (PEN 6871) IE 551, 2 octets, unsigned

       An identification number for the record that is user specifiable at 
       runtime and shared across all records in a run of the given program.

    exporterUniqueId, CERT (PEN 6871) IE 552, 2 octets, unsigned

       An identification number for the record that is randomly generated 
       at runtime and shared across all records in a run of the given program.

    subTemplateList, IE 292, variable length

       A subTemplateList consisting of Tombstone Access Templates (see below) 
       that specify when each program that supports tombstone timestamping 
       interacted with the tombstone record.

Tombstone Access Template

The following two Information Elements will be exported as a template within the subTemplateList of a Tombstone Options Template:

    exportingProcessId, IE 161, 4 octets, unsigned

       The identification number of the program that interacted with the record. 
       B<super_mediator> has and ID of 2.

    observationTimeSeconds, IE 322, 4 octets, dateTimeSeconds

       The UNIX timestamp of when the program interacted with the record.

To run super_mediator with the configuration file:

"super_mediator -c /usr/local/etc/super_mediator.conf"

To read a yaf(1) file and write delimited text to stdout:

"super_mediator -i yaffile.yaf -o - -m text"

To listen for connections from yaf(1) via TCP and write to rotating text files:

"super_mediator -i localhost --ipfix-port 18000 --ipfix-input TCP --out /tmp/mediator --rotate 120 --log /tmp/mediator.log -v"

To listen for connections from yaf(1) via UDP and send output to a rwflowpack process running on TCP port 18001:

"super_mediator -p 18000 --ipfix-input UDP -o localhost -m TCP --export-port 18001"

To only print the time and 5-tuple for each flow to stdout:

"super_mediator -i /myyaffile.yaf -o - -m TEXT --fields=18,19,0,1,4,5,6"

super_mediator requires libfixbuf 1.7.0 or later.

super_mediator will not create new file directories. All output directories must exist before you start super_mediator.

yaf presently encodes the ICMP type and code information into the destinationTransportPort information element for ICMP and ICMP6 flows. super_mediator running in TEXT output mode writes the type in the sourceTransportPort field and the ICMP code in the destinationTransportPort field.

Emily Sarneso and the CERT Network Situational Awareness Group Engineering Team <http://www.cert.org/netsa>. Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.

yaf(1), Spread Documentation at www.spread.org, and the following IETF Internet RFCs: Specification of the IPFIX Protocol for Exchange of IP Traffic Flow Information RFC 5101, Information Model for IP Flow Information Export RFC 5102, Export of Structured Data in RFC 6313.
14-May-2022 1.6.0

Search for    or go to Top of page |  Section 1 |  Main Index

Powered by GSP Visit the GSP FreeBSD Man Page Interface.
Output converted with ManDoc.