NAME

rwptoflow - Generate SiLK Flow records from packet data

SYNOPSIS

  rwptoflow [--plugin=PLUGIN [--plugin=PLUGIN ...]]
        [--active-time=YYYY/MM/DD:hh:dd:mm:ss.uuuuuu-YYYY/MM/DD:hh:dd:mm:ss.uuuuuu]
        [--flow-output=FLOW_PATH] [--packet-pass-output=PCKTS_PASS]
        [--packet-reject-output=PCKTS_REJECT]
        [--reject-all-fragments] [--reject-nonzero-fragments]
        [--reject-incomplete] [--set-sensorid=SCALAR]
        [--set-inputindex=SCALAR] [--set-outputindex=SCALAR]
        [--set-nexthopip=IP_ADDRESS] [--print-statistics]
        [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD] TCPDUMP_INPUT

  rwptoflow [--plugin=PLUGIN ...] --help

  rwptoflow --version

DESCRIPTION

rwptoflow attempts to generate a SiLK Flow record for every Ethernet IP IPv4 packet in the pcap(3) (tcpdump(1)) capture file TCPDUMP_INPUT. TCPDUMP_INPUT must contain data captured from an Ethernet datalink.

rwptoflow does not attempt to reassemble fragmented packets or to combine multiple packets into a single flow record. rwptoflow is a simple program that creates one SiLK Flow record for every IPv4 packet in TCPDUMP_INPUT. (For an alternate approach, consider using the rwp2yaf2silk(1) tool as described at the end of this section.)

rwptoflow will read from its standard input if TCPDUMP_INPUT is specified as "stdin". The SiLK Flow records are written to the specified flow-output file or to the standard output. The application will fail when attempting to read or write binary data from or to a terminal.

Packets outside of a user-specified active-time window can be ignored. Additional filtering on the TCPDUMP_INPUT can be performed by using tcpdump with an "expression" filter and piping tcpdump's output into rwptoflow.

In addition to generating flow records, rwptoflow can write pcap files containing the packets that it used to generate each flow, and/or the packets that were rejected. Note that packets falling outside the active-time window are ignored and are not written to the packet-reject-output.

Statistics of the number of packets read, rejected, and written can be printed.

rwptoflow will reject any packet that is not an IPv4 Ethernet packet and any packet that is too short to contain the Ethernet and IP headers. At the user's request, packets may be rejected when

they are fragmented---either the initial (zero-offset) fragment or a subsequent fragment
they have a non-zero fragment offset
they are not fragmented or they are the zero-fragment but the capture file does not contain enough information about the packet to set protocol-specific information---namely the ICMP type and code, the UDP source and destination ports, or the TCP source and destination ports and flags

Since the input packet formats do not contain some fields normally found in NetFlow data, rwptoflow provides a way to set those flow values in all packets. For example, it is possible to set the sensor-id manually for a tcpdump source, so that flow data can be filtered or sorted by that value later.

Alternative to rwptoflow

As mentioned above, rwptoflow is a simple program for processing Ethernet IP IPv4 packets. rwptoflow does not:

reassemble fragmented packets
support IPv6 packets
combine multiple packets into a single flow record
support any decoding of packets (e.g., 802.1q)

For these features (and others), you should use the yaf (1) application (<http://tools.netsa.cert.org/yaf/>) to read the pcap file and generate an IPFIX stream, and pipe the IPFIX stream into rwipfix2silk(1) to convert it to SiLK Flow records.

The rwp2yaf2silk(1) script makes this common usage more convenient by wrapping the invocation of yaf and rwipfix2silk. You give rwp2yaf2silk a pcap file and it writes SiLK Flow records.

By default, rwptoflow creates a flow record for every packet, fragments and all. You can almost force yaf to create a flow record for every packet: When you give yaf the --idle-timeout=0 switch, yaf creates a flow record for every complete packet and for each packet that it is able to completely reassemble from packet fragments. Any fragmented packets that yaf cannot reassemble are dropped.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--plugin=PLUGIN: Use the specified plug-in to ignore or reject packets or to modify the flow record that is generated from the packet. The switch may be repeated to load multiple plug-ins. See the "PLUG-IN SUPPORT" section below for details.
--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]-YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]: Ignore all packets whose time falls outside the specified range. The times must be specified to at least day precision. The start time is required; when the end-time is not present, it is treated as infinite. The end-time will be rounded-up to instant before the next time unit; i.e., an end-time of 2006/08/31:15 is treated as 2006/08/31:15:59:59.999999.
--flow-output=FLOW_PATH: Write the generated SiLK Flow records to the specified file at FLOW_PATH. When this switch is not provided, the flows are written to the standard output.
--packet-pass-output=PCKTS_PASS: For each generated SiLK Flow record, write the packet that generated the flow to the pcap file specified by PCKTS_PASS. Use "stdout" to write the packets to the standard output.
--packet-reject-output=PCKTS_REJECT: Write each packet that occurs within the active-time window but for which a SiLK Flow record was not generated to the pcap file specified by PCKTS_REJECT. Use "stdout" to write the packets to the standard output.
The packets that get written to this file may include packets that were shorter than that required to get the IP header, non-IPv4 packets, and packets that get treated as "reject" packets by the following switches.
--reject-all-fragments: Do not generate a SiLK Flow record for the packet when the packet is fragmented. This includes the initial (zero-offset) fragment and all subsequent fragments. If --packet-reject-output is specified, the packet will be written to that file.
--reject-nonzero-fragments: Do not generate a SiLK Flow record for the packet when the packet is fragmented unless this is the initial fragment. That is, reject all packets that have a non-zero fragmentation offset. Normally flow records are generated for these packets, but the ports and TCP flag information is set to zero. If --packet-reject-output is specified, the packet will be written to that file.
--reject-incomplete: Do not generate a SiLK Flow record for the packet when the packet's fragmentation-offset is zero yet the packet does not contain enough information to completely specify an ICMP, UDP, or TCP record (that is, the packet is too short to set the ICMP type and code, the UDP or TCP source or destination port, or the TCP flags). Normally, flow records are generated for these packets but the ports and TCP flag information is set to zero. This switch has no effect on packets where the protocol is not 1,6, or 17.
This switch does not imply --reject-nonzero-fragments; to indicate that all generated flow records must have valid port and TCP flag information, specify --reject-nonzero-fragments --reject-incomplete.
--set-sensorid=SCALAR: Set the sensor ID for all flows to SCALAR. SCALAR should be an integer value between 0 and 65534, inclusive. When not specified, the sensor ID is set to 65535.
--set-inputindex=SCALAR: Set the input SNMP index value for all flows to SCALAR. SCALAR should be an integer value between 0 and 65535, inclusive. When not specified, the SNMP input is set to 0.
--set-outputindex=SCALAR: Set the output SNMP index value for all flows to SCALAR. SCALAR should be an integer value between 0 and 65535, inclusive. When not specified, the SNMP output is set to 0.
--set-nexthopip=IP_ADDRESS: Set the next-hop IP address for all flows to IP_ADDRESS; IP_ADDRESS may be in its canonical form or an integer. When not specified, the next-hop IP is set to 0.0.0.0.
--print-statistics: Print a summary of the packets that were processed. This summary includes

the total number of packets read
the number that fell outside the time-window
the number that were too short to get the IP header
the number that were not IPv4
the number that were discarded by a plug-in
the total number of fragmented packets
the number of fragments where the offset was zero
the number of zero-offset packets that were incomplete
the number of flows written to the output

--note-add=TEXT: Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.
--note-file-add=FILENAME: Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.
--compression-method=COMP_METHOD: Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

none: Do not compress the output using an external library.
zlib: Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed.
lzo1x: Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.
snappy: Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0.
best: Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available. Only compress the output when writing to a file.

--help: Print the available options and exit. Options that add fields can be specified before --help so that the new options appear in the output.
--version: Print the version number and information about how SiLK was configured, then exit the application.

PLUG-IN SUPPORT

rwptoflow allows the user to provide additional logic to ignore or reject packets, or to modify the flow record that is generated from the packet. To do this, the user creates a plug-in that gets loaded at run-time by giving rwptoflow the --plugin switch with the path to the plug-in as the parameter to the switch.

A plug-in is a shared object file (a.k.a. dynamic library) that is compiled from C source code. The plug-in should have four subroutines defined:

setup(): is called when the object is first loaded. This is the place to initialize global variables to their default values. If the plug-in provides switches of its own, they must be registered in this subroutine.
initialize(): gets called after all options have been processed but before any packets are read from the input. If this subroutine does not return 0, the application will quit.
ptoflow(): will be called for every packet that rwptoflow is able to convert into a flow record just before the flow record is written. This subroutine will not see packets that are short or that are not IPv4; it will also not see fragmented packets if --reject-all-fragments is specified.
The "ptoflow()" function is called with two parameters:

a pointer to the "rwRec" object that rwptoflow created from the packet. The subroutine may modify the record as it sees fit.

a void pointer that the function may cast to a pointer to the C structure:

  typedef struct _sk_pktsrc_t {
      /* the source of the packets */
      pcap_t                     *pcap_src;
      /* the pcap header as returned from pcap_next() */
      const struct pcap_pkthdr   *pcap_hdr;
      /* the packet as returned from pcap_next() */
      const u_char               *pcap_data;
  } sk_pktsrc_t;

This structure gives the user access to all the information about the packet.

The following return values from "ptoflow()" determines whether rwptoflow writes the flow and the packet:

0: Write the flow record to the flow-output and the packet to the PCKTS_PASS unless another plug-in instructs otherwise.
1: Write the flow record to the flow-output and the packet to the PCKTS_PASS immediately; do not call the "ptoflow()" routine on any other plug-in.
2: Treat the packet as a reject: Do not write the flow record; write the packet to the PCKTS_REJECT immediately; do not call the "ptoflow()" routine on any other plug-in.
3: Ignore the packet immediately: Do not write the flow record nor the packet; do not call the "ptoflow()" routine on any other plug-in.

If "ptoflow()" returns any other value, the rwptoflow application will terminate with an error.

teardown(): is called as the application exits. The user can use this routine to print results and to free() any data structures that were used.

rwptoflow uses the following rules to find the plug-in: When PLUGIN contains a slash ("/"), rwptoflow assumes the path to PLUGIN is correct. Otherwise, rwptoflow will attempt to find the file in $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these directories parallel to the application's directory: lib/silk, share/lib, and lib. If rwptoflow does not find the file, it assumes the plug-in is in the current directory. To force rwptoflow to look in the current directory first, specify --plugin=./PLUGIN. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwptoflow prints status messages to the standard error as it tries to open each of its plug-ins.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line.

Given the packet capture file data.pcap, convert it to a SiLK flow file, data.rw, and copy the packets that rwptoflow understands to the file good.pcap:

 $ rwptoflow data.pcap --packet-pass=good.pcap --flow-out=data.rw

Use rwfilter(1) to partition the SiLK Flows records, writing those records whose source IPs are found in the IPset file sip.set to filtered.rw:

 $ rwfilter --sipset=sip.set --pass=filtered.rw  data.rw

Use rwpmatch(1) to match the capture file, good.pcap, against the filtered SiLK file, in affect generating a capture file which has been filtered by sip.set:

 $ rwpmatch --flow-file=filtered.rw good.pcap > filtered.pcap

ENVIRONMENT

SILK_PLUGIN_DEBUG: When set to 1, rwptoflow print status messages to the standard error as it tries to open each of its plug-ins.
SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_COMPRESSION_METHOD: This environment variable is used as the value for --compression-method when that switch is not provided. Since SiLK 3.13.0.
SILK_PATH: This environment variable gives the root of the install tree. When searching for a plug-in, rwptoflow may use this environment variable.

NOTES

SiLK supports millisecond timestamps. When reading packets whose timestamps have finer precision, the times are truncated at the millisecond position.

The mergecap(1) or rwpdedupe(1) programs can be used to join multiple tcpdump capture files in order to convert into a single flow file.