The SiLK analysis suite is a collection of command-line tools for
processing SiLK Flow records created by the SiLK packing system.
These tools read binary files containing SiLK Flow records and
partition, sort, and count these records. The most important analysis
tool is rwfilter(1), an application for querying the central data
repository for SiLK Flow records that satisfy a set of filtering
options. The tools are intended to be combined in various ways to
perform an analysis task. A typical analysis uses UNIX pipes and
intermediate data files to share data between invocations of tools.
The tools, configuration files, and plug-in modules that make up the
analysis tools are listed below, roughly grouped by functionality.
Filtering, sorting, and display
rwfilter(1) partitions SiLK Flow records into one or more pass
and/or fail output streams. rwfilter is the primary tool for
pulling flows from the data store.
silk.conf(5) is the configuration file naming the Classes, Types,
and Sensors available at your installation.
rwsort(1) sorts SiLK Flow records using a user-specified key
comprised of record attributes, and writes the records to the named
output path or to the standard output. Users can define new key
fields using plug-ins written in C or PySiLK.
rwcut(1) prints the attributes of SiLK Flow records in a delimited,
columnar, human-readable format. Users can define new printable
attributes using plug-ins written in C or PySiLK.
SiLK Python Extension
pysilk(3). PySiLK, the SiLK Python extension, allows one to read,
manipulate, and write SiLK Flow records, IPsets, and Bags from within
Python. PySiLK may be used in a stand-alone Python program or to
write plug-ins for several SiLK applications. This document describes
the objects, methods, and functions that PySiLK provides. The next
entry describes using PySiLK from within a plug-in.
silkpython(3). The SiLK Python plug-in provides a way to use
PySiLK to define new partitioning rules for rwfilter(1), new key
fields for rwcut(1), rwgroup(1), and rwsort(1), and new key
or value fields for rwstats(1) and rwuniq(1).
Counting and statistics
rwuniq(1) bins (groups) SiLK Flow records by a user-specified key
comprised of record attributes and prints the total byte, packet,
and/or flow counts for each bin. rwuniq can also print distinct
source IP and destination IP counts. Users can define new key fields
and value fields using plug-ins written in C or PySiLK.
rwcount(1) summarizes SiLK Flow records across time, producing
textual output with counts of bytes, packets, and flow records for
each time bin.
rwstats(1) summarizes SiLK Flow records by a user-specified key
comprised of record attributes, computes values from the flow records
that match each key, sorts the results by the value to generate a
Top-N or Bottom-N list, and prints the results. Users can define new
key fields and value fields using plug-ins written in C or PySiLK.
rwtotal(1) summarizes SiLK Flow records by a specified key and
prints the sum of the byte, packet, and flow counts for flows matching
rwaddrcount(1) summarizes SiLK flow records by the source or
destination IP and prints the byte, packet, and flow counts for each
IPset, Bag, and Prefix Map manipulation
rwset(1) reads SiLK Flow records and generates binary IPset file(s)
containing the source IP addresses or destination IP addresses seen on
the flow records.
rwsetbuild(1) reads (textual) IP addresses in dotted-quad or CIDR
notation from an input file or from the standard input and writes a
binary IPset file.
rwsetcat(1) prints the contents of a binary IPset file as text.
Additional information about the IPset file can be printed.
rwsettool(1) performs union, intersection, difference, and sampling
functions on the input IPset files, generating a new IPset file.
rwsetmember(1) determines whether the IP address specified on the
command line is contained in an IPset.
rwbag(1) reads SiLK Flow records and builds binary Bag(s)
containing key-count pairs. An example is a Bag containing the sum of
the byte counts for each source port seen on the flow records.
rwbagbuild(1) creates a binary Bag file from a binary IPset file or
from a textual input file.
rwbagcat(1) prints binary Bag files as text.
rwbagtool(1) performs operations (e.g., addition, subtraction) on
binary Bag files and produces a new Bag file.
rwpmapbuild(1) reads textual input and creates a binary prefix map
file for use with the Address Type (addrtype(3)) and Prefix Map
rwpmapcat(1) prints information about a prefix map file as text.
By default, prints each IP range in the prefix map and its label.
rwpmaplookup(1) finds information about specific IP address(es) or
protocol/port pair(s) in a binary prefix map file and prints the
result as text.
rwipaimport(1) imports a SiLK IPset, Bag, or Prefix Map file into
the IP Address Association (IPA <http://tools.netsa.cert.org/ipa/>)
rwipaexport(1) exports a set of IP addresses from the IP Address
Association (IPA) library to a SiLK IPset, Bag, or Prefix Map.
IP and port labeling files
addrtype(3). The Address Type file provides a way to map an IPv4
address to an integer denoting the IP as internal, external, or
ccfilter(3). The Country Code file provides a mapping from an IPv4
address to two-letter, lowercase abbreviation of the country what that
IP address is located. The abbreviations used by the Country Code
utility are those used by the Root-Zone Whois Index (see for example
pmapfilter(3). Prefix map files provide a way to map field
values to string labels based on a user-defined map file. The map
file is created by rwpmapbuild(1).
Run time plug-ins
flowrate(3). The flowrate plug-in, which must be loaded
explicitly, adds switches and fields to compute packets/second,
bytes/second, bytes/packet, payload-bytes, and payload-bytes/second.
int-ext-fields(3). The internal/external plug-in makes available
fields containing internal and external IPs and ports (int-ip, ext-ip,
int-port, and ext-port).
ipafilter(3). The IPA (IP Association) plug-in works with
rwfilter to partition flows based on data in an IPA data store.
rwfilter will automatically load this plug-in if it is available.
The plug-in requires that SiLK be compiled with IPA support
silk-plugin(3) describes how to create and compile a new SiLK
plug-in using C.
Record grouping and masking
rwgroup(1) groups SiLK flow records by a user-specified key
comprised of record attributes, labels the records with a group ID
that is stored in the next-hop IP field, and writes the resulting
flows to the specified output path or to the standard output.
rwgroup requires that its input is sorted.
rwmatch(1) matches (mates) records as queries and responses and
marks mated records with an ID that is stored in the next-hop IP
field. rwmatch requires that its input is sorted.
rwnetmask(1) reads SiLK Flow records, zeroes the least significant
bits of the source-, destination-, and/or next-hop-IP address(es), and
writes the resulting records to the named output path or to the
Packet and external flow-format processing
rwp2yaf2silk(1) converts a packet capture (pcap(3)) file---such
as a file produced by tcpdump(1)---to a single file of SiLK Flow
records. rwp2yaf2silk assumes that the yaf(1)
(<http://tools.netsa.cert.org/yaf/>) and rwipfix2silk(1) commands
are available on your system as it is a simple Perl wrapper around
rwipfix2silk(1) converts a stream of IPFIX (Internet Protocol Flow
Information eXport) records to the SiLK Flow record format.
rwsilk2ipfix(1) converts a stream of SiLK Flow records to an IPFIX
(Internet Protocol Flow Information eXport) format.
rwpcut(1) reads a packet capture file and print its contents in a
textual form similar to that produced by rwcut.
rwpdedupe(1) detects and eliminates duplicate records from multiple
packet capture input files. See also rwdedupe(1).
rwpmatch(1) filters a packet capture file by writing only
packets whose five-tuple and timestamp match corresponding records in
a SiLK Flow file.
rwptoflow(1) reads a packet capture file and generates a SiLK Flow
record for every packet.
rwpdu2silk(1) creates a stream of SiLK Flow records from a file
containing NetFlow v5 PDU records.
rwscan(1) attempts to detect scanning activity from SiLK Flow
records. rwscan can produce files that can be loaded into a
database and queried with rwscanquery.
rwscanquery(1) queries the scan database which has been populated
from database load files generated by rwscan.
rwcat(1) reads SiLK Flow records from the files named on the
command line, or from the standard input when no files are provided,
and writes the SiLK records to the specified output file or to the
standard output if it is not connected to a terminal.
rwappend(1) appends the SiLK Flow records contained in the second
through final file name arguments to the records contained in the
first file name argument.
rwcombine(1) reads SiLK Flow records from files named on the
command line or from the standard input. For records where the
attributes field contains the flow timed-out flag, rwcombine
attempts to find the record with the corresponding continuation
flag set and combine those records into a single flow. rwcombine
writes the results to the named output file or to the standard output.
rwcompare(1) determines whether two SiLK Flow files contain the
same flow records.
rwdedupe(1) reads SiLK Flow records from files named on the command
line or from the standard input and writes the records to the named
output path or to the standard output, removing any duplicate flow
records. Note that rwdedupe will reorder the records as part of its
rwrandomizeip(1) generates a new SiLK Flow file by substituting a
pseudo-random IP address for the source and destination IP addresses
in given input file.
rwrecgenerator(1) generates SiLK Flow records using a pseudo-random
number generator; these records can be used to test SiLK applications.
rwsplit(1) reads SiLK Flow records and generates a set of sub-files
from the input. The sub-files can be limited by flow-, byte-, or
packet-counts, or by unique IP count. In addition, the sub-file may
contain all the flows or only a sample of them.
rwswapbytes(1) generates a new SiLK Flow file by changing the byte
order of the records in a given input SiLK Flow file.
rwfileinfo(1) prints information (type, version, etc.) about a SiLK
Flow, IPset, Bag, or Prefix Map file.
rwsiteinfo(1) prints information about the sensors, classes, and
types specified in the silk.conf(5) file.
rwtuc(1) generates SiLK flow records from textual input; the input
should be in a form similar to what rwcut(1) generates.
rwfglob(1) prints to the standard output the list of files that
rwfilter would normally process for a given set of file selection
num2dot(1) reads delimited text from the standard input, converts
integer values in the specified column(s) (default first column) to
dotted-decimal IP address, and prints the result to the standard
rwgeoip2ccmap(1) reads the MaxMind GeoIP database and creates the
country code mapping file that can be used by SiLK (see
rwidsquery(1) invokes rwfilter to find flow records matching
rwresolve(1) reads delimited text from the standard input, attempts
to resolve the IP addresses in the specified column(s) to host names,
and prints the result to the standard output.
silk_config(1) prints information about how SiLK was compiled; this
information can be used to compile and link other files and programs
against the SiLK header files and libraries.
mapsid(1) maps between sensor names and sensor IDs using the values
specified in the silk.conf(5) file. mapsid is deprecated as of
SiLK 3.0.0, and it will be removed in the SiLK 4.0 release. This
functionality is available in rwsiteinfo(1).
rwguess(8) reads a file containing NetFlow v5 PDU records and
prints the SNMP interfaces that are used most often and the number of
records seen for each interface. rwguess is deprecated as of SiLK
3.8.3, and it will be removed in the SiLK 4.0 release. Similar
functionality is available using a combination of rwpdu2silk(1),
rwstats(1), and rwuniq(1).
rwip2cc(1) maps a (textual) list of IP addresses to their country
code. rwip2cc is deprecated as of SiLK 3.0.0, and it will be
removed in the SiLK 4.0 release. This functionality is available in