NAME

SiLK - the System for Internet-Level Knowledge

DESCRIPTION

SiLK is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks. The SiLK tool suite supports the efficient collection, storage, and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.

A SiLK installation consists of two categories of applications: the analysis suite and the packing system.

Analysis Suite

The SiLK analysis suite is a collection of command-line tools for processing SiLK Flow records created by the SiLK packing system. These tools read binary files containing SiLK Flow records and partition, sort, and count these records. The most important analysis tool is rwfilter(1), an application for querying the central data repository for SiLK Flow records that satisfy a set of filtering options. The tools are intended to be combined in various ways to perform an analysis task. A typical analysis uses UNIX pipes and intermediate data files to share data between invocations of tools.

The tools, configuration files, and plug-in modules that make up the analysis tools are listed below, roughly grouped by functionality.

Filtering, Sorting, and Display

rwfilter(1) partitions SiLK Flow records into one or more 'pass' and/or 'fail' output streams. rwfilter is the primary tool for pulling flows from the data store.

silk.conf(5) is the configuration file naming the Classes, Types, and Sensors available at your installation.

rwsort(1) sorts SiLK Flow records using a user-specified key comprised of record attributes, and writes the records to the named output path or to the standard output. Users may define new key fields using plug-ins written in C or PySiLK.

rwcut(1) prints the attributes of SiLK Flow records in a delimited, columnar, human-readable format. Users may define new printable attributes using plug-ins written in C or PySiLK.

SiLK Python Extension

pysilk(3). PySiLK, the SiLK Python extension, allows one to read, manipulate, and write SiLK Flow records, IPsets, and Bags from within Python. PySiLK may be used in a stand-alone Python program or to write plug-ins for several SiLK applications. This document describes the objects, methods, and functions that PySiLK provides. The next entry describes using PySiLK from within a plug-in.

silkpython(3). The SiLK Python plug-in provides a way to use PySiLK to define new partitioning rules for rwfilter(1), new key fields for rwcut(1), rwgroup(1), and rwsort(1), and new key or value fields for rwstats(1) and rwuniq(1).

Counting, Grouping, and Mating

rwuniq(1) bins (groups) SiLK Flow records by a user-specified key comprised of record attributes and prints the total byte, packet, and/or flow counts for each bin. rwuniq may also print distinct source IP and destination IP counts. Users may define new key fields and value fields using plug-ins written in C or PySiLK.

rwcount(1) summarizes SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin.

rwstats(1) summarizes SiLK Flow records by a user-specified key comprised of record attributes, computes values from the flow records that match each key, sorts the results by the value to generate a Top-N or Bottom-N list, and prints the results. Users may define new key fields and value fields using plug-ins written in C or PySiLK.

rwtotal(1) summarizes SiLK Flow records by a specified key and prints the sum of the byte, packet, and flow counts for flows matching the key.

rwaddrcount(1) summarizes SiLK flow records by the source or destination IP and prints the byte, packet, and flow counts for each IP.

rwgroup(1) groups SiLK flow records by a user-specified key comprised of record attributes, labels the records with a group ID that is stored in the next-hop IP field, and writes the resulting flows to the specified output path or to the standard output. rwgroup requires that its input is sorted.

rwmatch(1) matches (mates) records as queries and responses and marks mated records with an ID that is stored in the next-hop IP field. rwmatch requires that its input is sorted.

IPsets, Bags, Aggregate Bags, and Prefix Maps

An IPset is a data structure and a binary file format that contains a list of IP addresses where each IP appears once (a mathematical set).

A Bag is a data structure and a binary file format where a key is mapped to a counter (similar to a hash table or Python dictionary). The key is either a 32-bit number or an IPv6 address, and the counter is a 64-bit number. Usually the key represents an aspect of a flow record (an IP address, a port number, the protocol) and the counter is a volume (the number of flow records, the sum of the packet counts) for the flow records that match that key.

An Aggregate Bag is similar to a Bag except the key and/or the counter may be comprised of multiple fields. Aggregate Bags were introduced in SiLK 3.15.0.

A prefix map is a data structure and file format that maps every IP address to string. An example prefix map gives the two-letter country code for any IP address.

rwset(1) reads SiLK Flow records and generates binary IPset file(s) containing the source IP addresses or destination IP addresses seen on the flow records.

rwsetbuild(1) reads (textual) IP addresses in dotted-quad or CIDR notation from an input file or from the standard input and writes a binary IPset file.

rwsetcat(1) prints the contents of a binary IPset file as text. Additional information about the IPset file may be printed.

rwsettool(1) performs union, intersection, difference, and sampling functions on the input IPset files, generating a new IPset file.

rwsetmember(1) determines whether the IP address specified on the command line is contained in an IPset.

rwbag(1) reads SiLK Flow records and builds binary Bag(s) containing key-count pairs. An example is a Bag containing the sum of the byte counts for each source port seen on the flow records.

rwbagbuild(1) creates a binary Bag file from a binary IPset file or from a textual input file.

rwbagcat(1) prints binary Bag files as text.

rwbagtool(1) performs operations (e.g., addition, subtraction) on binary Bag files and produces a new Bag file.

rwaggbag(1) reads SiLK Flow records and builds a binary Aggregate Bag containing key-count pairs. An example is a Aggregate Bag containing the sum of the byte counts for each source port seen on the flow records. Since SiLK 3.15.0.

rwaggbagbuild(1) creates a binary Aggregate Bag file from a textual input file. Since SiLK 3.15.0.

rwaggbagcat(1) prints binary Aggregate Bag files as text. Since SiLK 3.15.0.

rwaggbagtool(1) performs operations (e.g., addition, subtraction) on binary Aggregate Bag files and produces a new Aggregate Bag file. Since SiLK 3.15.0.

rwpmapbuild(1) reads textual input and creates a binary prefix map file for use with the Address Type (addrtype(3)) and Prefix Map (pmapfilter(3)) utilities.

rwpmapcat(1) prints information about a prefix map file as text. By default, prints each IP range in the prefix map and its label.

rwpmaplookup(1) finds information about specific IP address(es) or protocol/port pair(s) in a binary prefix map file and prints the result as text.

rwipaimport(1) imports a SiLK IPset, Bag, or Prefix Map file into the IP Address Association (IPA <http://tools.netsa.cert.org/ipa/>) library.

rwipaexport(1) exports a set of IP addresses from the IP Address Association (IPA) library to a SiLK IPset, Bag, or Prefix Map.

IP and Port Labeling Files

addrtype(3). The Address Type file provides a way to map an IPv4 address to an integer denoting the IP as internal, external, or non-routable.

ccfilter(3). The Country Code file provides a mapping from an IP address to two-letter, lowercase abbreviation of the country what that IP address is located. The abbreviations used by the Country Code utility are those defined by ISO 3166-1 (see for example <https://www.iso.org/iso-3166-country-codes.html> or <https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2>).

pmapfilter(3). Prefix map files provide a way to map field values to string labels based on a user-defined map file. The map file is created by rwpmapbuild(1).

Run Time Plug-Ins

To use most of these plug-ins, the plug-in must be explicitly loaded into an application by using the application's --plugin switch and giving the plug-in's library name or path as the argument. For a plug-in named NAME, the library is typically named NAME.so.

app-mismatch(3). The application-mismatch plug-in helps to find services running on unusual or non-typical ports by causing rwfilter(1) to only pass a flow record when the record's application field is non-zero and its value is different than that in the source port and destination port fields.

conficker-c(3). The conficker-c plug-in was written in March 2009 to detect traffic that matches the signature of the .C variant of the Conficker worm.

cutmatch(3). The cutmatch plug-in creates a field in rwcut(1) that provides a more user-friendly representation of the match parameter value that rwmatch(1) writes into a SiLK Flow record's next hop IP field.

flowkey(3). The flowkey plug-in adds a switch and a field that computes a 32-bit hash for a flow record using the same algorithm as YAF uses for its flow key utility getFlowKeyHash(1). Since SiLK 3.15.0.

flowrate(3). The flowrate plug-in adds switches and fields to compute packets/second, bytes/second, bytes/packet, payload-bytes, and payload-bytes/second.

int-ext-fields(3). The internal/external plug-in makes available fields containing internal and external IPs and ports (int-ip, ext-ip, int-port, and ext-port). It can be used to print, sort by, or group by the internal or external IP or port, which is useful when a single flow file contains flows in multiple directions. Since SiLK 3.0.0.

ipafilter(3). The IPA (IP Association) plug-in works with rwfilter to partition flows based on data in an IPA data store. rwfilter will automatically load this plug-in if it is available. The plug-in requires that SiLK be compiled with IPA support (<http://tools.netsa.cert.org/ipa/>).

silk-plugin(3) describes how to create and compile a new SiLK plug-in using C.

Packet and IPFIX Processing

These tools operate on packet capture (pcap(3)) files, IPFIX files, or files of NetFlow v5 data.

rwp2yaf2silk(1) converts a packet capture (pcap(3)) file---such as a file produced by tcpdump(1)---to a single file of SiLK Flow records. rwp2yaf2silk assumes that the yaf(1) (<http://tools.netsa.cert.org/yaf/>) and rwipfix2silk(1) commands are available on your system as it is a simple Perl wrapper around those commands.

rwipfix2silk(1) converts a stream of IPFIX (Internet Protocol Flow Information eXport) records to the SiLK Flow record format.

rwsilk2ipfix(1) converts a stream of SiLK Flow records to an IPFIX (Internet Protocol Flow Information eXport) format.

rwpcut(1) reads a packet capture file and print its contents in a textual form similar to that produced by rwcut.

rwpdedupe(1) detects and eliminates duplicate records from multiple packet capture input files. See also rwdedupe(1).

rwpmatch(1) filters a packet capture file by writing only packets whose five-tuple and timestamp match corresponding records in a SiLK Flow file.

rwptoflow(1) reads a packet capture file and generates a SiLK Flow record for every packet.

rwpdu2silk(1) creates a stream of SiLK Flow records from a file containing NetFlow v5 PDU records.

Scan Detection

rwscan(1) attempts to detect scanning activity from SiLK Flow records. rwscan can produce files that may be loaded into a database and queried with rwscanquery.

rwscanquery(1) queries the scan database which has been populated from database load files generated by rwscan.

Flow File Utilities

These utility applications operate on SiLK Flow files.

rwcat(1) reads SiLK Flow records from the files named on the command line, or from the standard input when no files are provided, and writes the SiLK records to the specified output file or to the standard output if it is not connected to a terminal.

rwappend(1) appends the SiLK Flow records contained in the second through final file name arguments to the records contained in the first file name argument.

rwcombine(1) reads SiLK Flow records from files named on the command line or from the standard input. For records where the attributes field contains the flow timed-out flag, rwcombine attempts to find the record with the corresponding continuation flag set and combine those records into a single flow. rwcombine writes the results to the named output file or to the standard output. Since SiLK 3.9.0.

rwcompare(1) determines whether two SiLK Flow files contain the same flow records.

rwdedupe(1) reads SiLK Flow records from files named on the command line or from the standard input and writes the records to the named output path or to the standard output, removing any duplicate flow records. Note that rwdedupe will reorder the records as part of its processing.

rwnetmask(1) reads SiLK Flow records, zeroes the least significant bits of the source-, destination-, and/or next-hop-IP address(es), and writes the resulting records to the named output path or to the standard output.

rwrandomizeip(1) generates a new SiLK Flow file by substituting a pseudo-random IP address for the source and destination IP addresses in given input file.

rwrecgenerator(1) generates SiLK Flow records using a pseudo-random number generator; these records may be used to test SiLK applications. Since SiLK 3.6.0.

rwsplit(1) reads SiLK Flow records and generates a set of sub-files from the input. The sub-files may be limited by flow-, byte-, or packet-counts, or by unique IP count. In addition, the sub-file may contain all the flows or only a sample of them.

rwswapbytes(1) generates a new SiLK Flow file by changing the byte order of the records in a given input SiLK Flow file.

Utilities

rwfileinfo(1) prints information (type, version, etc.) about a SiLK Flow, IPset, Bag, or Prefix Map file.

rwsiteinfo(1) prints information about the sensors, classes, and types specified in the silk.conf(5) file.

rwtuc(1) generates SiLK flow records from textual input; the input should be in a form similar to what rwcut (1) generates.

rwfglob(1) prints to the standard output the list of files that rwfilter would normally process for a given set of file selection switches.

num2dot(1) reads delimited text from the standard input, converts integer values in the specified column(s) (default first column) to dotted-decimal IP address, and prints the result to the standard output.

rwgeoip2ccmap(1) reads the MaxMind GeoIP database and creates the country code mapping file that may be used by SiLK (see ccfilter(3)).

rwidsquery(1) invokes rwfilter to find flow records matching Snort signatures.

rwresolve(1) reads delimited text from the standard input, attempts to resolve the IP addresses in the specified column(s) to host names, and prints the result to the standard output.

silk_config(1) prints information about how SiLK was compiled; this information may be used to compile and link other files and programs against the SiLK header files and libraries.

Deprecated Tools

These tools are deprecated. Their functionality is available in other applications.

mapsid(1) maps between sensor names and sensor IDs using the values specified in the silk.conf(5) file. mapsid is deprecated as of SiLK 3.0.0, and it will be removed in the SiLK 4.0 release. This functionality is available in rwsiteinfo(1).

rwguess(8) reads a file containing NetFlow v5 PDU records and prints the SNMP interfaces that are used most often and the number of records seen for each interface. rwguess is deprecated as of SiLK 3.8.3, and it will be removed in the SiLK 4.0 release. Similar functionality is available using a combination of rwpdu2silk(1), rwstats(1), and rwuniq(1).

rwip2cc(1) maps a (textual) list of IP addresses to their country code. rwip2cc is deprecated as of SiLK 3.0.0, and it will be removed in the SiLK 4.0 release. This functionality is available in rwpmaplookup(1).

Packing System

The SiLK Packing System is comprised of daemon applications that collect flow records (IPFIX flows from yaf(1) or NetFlow v5 or v9 PDUs from a router), convert the records to the SiLK flow format, categorize the flows as incoming or outgoing, and write the records to their final destination in binary flat files for use by the analysis suite. Files are organized in a time-based directory hierarchy with files covering each hour at the leaves.

The tools, configuration files, and plug-ins that comprise the SiLK Packing System are:

flowcap(8) listens to flow generators (devices which produce network flow data) and stores the data in temporary files prior to transferring the files to a remote machine for processing by rwflowpack.

rwflowpack(8) reads flow data either directly from a flow generator or from files generated by flowcap, converts the data to the SiLK flow record format, categorizes the flow records according to rules loaded from a packing-logic plug-in, and writes the records either to hourly flat-files organized in a time-based directory structure or to files for transfer to a remote machine for processing by rwflowappend.

rwflowappend(8) watches a directory for files containing small numbers of SiLK flow records and appends those records to hourly files organized in a time-based directory tree.

rwsender(8) watches an incoming directory for files, moves the files into a processing directory, and transfers the files to one or more rwreceiver processes. Either rwsender or rwreceiver may act as the server (i.e., listen for incoming network connections) with the other acting as the client.

rwreceiver(8) accepts files transferred from one or more rwsender processes and stores them in a destination directory. Either rwsender or rwreceiver may act as the server with the other acting as the client.

rwpollexec(8) monitors a directory for incoming files and runs a user-specified command on each file.

rwpackchecker(8) reads SiLK Flow records and checks for unusual patterns that may indicate data file corruption.

sensor.conf(5) is a configuration file for sensors and probes used by rwflowpack and flowcap.

packlogic-twoway(3) is one of the plug-ins available that describe a set of rules (the packing-logic) that rwflowpack may use when categorizing flow records as incoming or output.

packlogic-generic(3) is one of the plug-ins available that describe a set of rules (the packing-logic) that rwflowpack may use when categorizing flow records as incoming or output.

ENVIRONMENT

The following environment variables affect the tools in the SiLK tool suite. The variables are listed alphabetically. (Additional environment variables that are specific to a tool are documented on the tool's manual page.)

PAGER: The applications that support paging their output use the value in this environment variable when the SILK_PAGER environment variable is not set and the application's --pager switch is not used.
PYTHONPATH: The Python modules and library files required to use PySiLK from rwfilter(1), rwcut(1), rwsort(1), and rwuniq(1) as well as from Python itself are installed under SiLK's installation tree by default. It may be necessary to set or modify the PYTHONPATH environment variable so Python can find these files. For information on using the PySiLK module, see silkpython(3) as well as the SiLK in Python handbook.
PYTHONVERBOSE: If the SiLK Python extension or plug-in fails to load, setting this environment variable to a non-empty string may help you debug the issue.
RWRECEIVER_TLS_PASSWORD: Used by rwreceiver(8), this variable specifies the password to use to decrypt the PKCS#12 file specified in the --tls-pkcs12 switch.
RWSENDER_TLS_PASSWORD: Used by rwsender(8), this variable specifies the password to use to decrypt the PKCS#12 file specified in the --tls-pkcs12 switch.
SILK_ADDRESS_TYPES: This environment variable allows the user to specify the address types mapping file used by the fields and switches specified in the addrtype(3) manual page. The value may be a complete path or a file relative to SILK_PATH. See the "FILES" section for standard locations of this file.
SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value (other than 0) removes this restriction.
SILK_COMPRESSION_METHOD: For most tools that implement the --compression-method switch, this environment variable is used as the value for that switch when it is not provided. Since SiLK 3.13.0.
SILK_CONFIG_FILE: This environment variable contains the location of the site configuration file, silk.conf(5). This variable has precedence over all methods of finding the site file except for the --site-config-file switch on an application. For additional locations where site configuration file may reside, see the "FILES" section.
SILK_COUNTRY_CODES: This environment variable allows the user to specify the country code mapping file used by the fields and switches specified in the ccfilter(3) manual page. The value may be a complete path or a file relative to SILK_PATH. See the "FILES" section for standard locations of this file.
SILK_DATA_ROOTDIR: This variable gives the root of directory tree where the data store of SiLK Flow files is maintained, overriding the location that is compiled into the tools (/data). The rwfilter(1) and rwfglob(1) tools use this value when selecting which flow files to process unless the user passes the --data-rootdir switch to the application. In addition, the SiLK tools search for the site configuration file, silk.conf, in this directory.
SILK_ICMP_SPORT_HANDLER: Modifies how "buggy" ICMP SiLK flow records are handled. ICMP type and code are normally encoded in the destination port field. Prior to SiLK 3.4.0, a bug existed when processing IPFIX bi-flow ICMP records where the type and code of the second records were stored in the source port. SiLK 3.4.0 attempts to work-around this bad encoding by modifying the buggy ICMP SiLK Flow records as they are initially read. However, the change in SiLK 3.4.0 removes a previous work-around designed to fix issues with SiLK Flow records collected prior to SiLK 0.8.0 that originated as NetFlow v5 PDUs from some types of Cisco routers. The ICMP records from these Cisco routers encoded the type and code in the source port, but the bytes were swapped from the normal encoding. When the SILK_ICMP_SPORT_HANDLER environment variable is set to "none", all work-arounds for buggy ICMP records are disabled and the source and destination ports remain unchanged.
SILK_IPSET_RECORD_VERSION: For the IPset family of tools, this environment variable is used as the default value for the --record-version switch when the switch is not provided on the command line. The variable is also used by rwbagtool(1) and rwaggbagtool(1) when writing an IPset file. Since SiLK 3.7.0.
SILK_IPV6_POLICY: For tools that implement the --ipv6-policy switch, this environment variable is used as the value for that switch when it is not provided.
SILK_IP_FORMAT: For tools that implement the --ip-format switch, this environment variable is used as the value for that switch when it is not provided. Since SiLK 3.11.0.
SILK_LOGSTATS: This environment variable is currently an alias for the SILK_LOGSTATS_RWFILTER environment variable described below. The ability to log invocations may be extended to other SiLK tools in future releases.
SILK_LOGSTATS_DEBUG: If the environment variable is set to a non-empty value, rwfilter(1) prints messages to the standard error about the SILK_LOGSTATS value being used and either the reason why the value cannot be used or the arguments to the external program being executed.
SILK_LOGSTATS_RWFILTER: When set to a non-empty value, rwfilter(1) treats the value as the path to a program to execute with information about this rwfilter invocation. Its purpose is to provide the SiLK administrator with information on how the SiLK tool set is being used.
SILK_PAGER: When this variable is set to a non-empty string, most of the applications that produce textual output (e.g., rwcut(1)) automatically invoke this program to display their output a screen at a time. If set to an empty string, no paging of the output is performed. The PAGER variable is checked when this variable is not set. The --pager switch on an application overrides this value.
SILK_PATH: This environment variable gives the root of the directory tree where the tools are installed. As part of its search for configuration files and plug-ins, a SiLK application may use this variable. See the "FILES" section for details.
SILK_PLUGIN_DEBUG: When this variable is set to a non-empty value, an application that supports plug-ins prints status messages to the standard error as it tries to locate and open each of its plug-ins.
SILK_PYTHON_TRACEBACK: If a Python plug-in encounters a Python-related error and this environment variable is set to a non-empty value, the application prints the error's traceback information to the standard error.
SILK_RWFILTER_THREADS: This variable sets the number of threads rwfilter(1) uses while reading input files or files selected from the data store.
SILK_TEMPFILE_DEBUG: When set to 1, the library that manages temporary files for rwcombine(1), rwdedupe(1), rwsort(1), rwstats(1), and rwuniq(1) prints debugging messages to the standard error as it creates, re-opens, and removes temporary files.
SILK_TIMESTAMP_FORMAT: For tools that implement the --timestamp-format switch, this environment variable is used as the value for that switch when it is not provided. Since SiLK 3.11.0.
SILK_TMPDIR: This variable is used by tools that write temporary files (e.g., rwsort(1)) as the directory in which to store those files. When this variable is not set, the value of the TMPDIR variable is checked. The --temp-directory switch on an application overrides this value.
SILK_UNIQUE_DEBUG: When set to 1, the binning engine used by rwstats(1) and rwuniq(1) prints debugging messages to the standard error.
TMPDIR: When this variable is set and SILK_TMPDIR is not set, temporary files are created in this directory. The value given to an application's --temp-directory switch takes precedence over both variables.
TZ: When a SiLK installation is built to use the local timezone (to determine if this is the case, check the "Timezone support" value in the output from the --version switch on most SiLK applications), the value of the TZ environment variable determines the timezone in which timestamps are displayed and parsed. If the TZ environment variable is not set, the default timezone is used. Setting TZ to 0 or to the empty string causes timestamps to be displayed in and parsed as UTC. The value of the TZ environment variable is ignored when the SiLK installation uses UTC unless the user requests use of the local timezone via a tool's --timestamp-format switch. For system information on the TZ variable, see tzset(3) or environ(7).

FILES

The following file and directory locations are used by SiLK tools. A dollar sign preceding a name enclosed in braces (e.g., "${SILK_PATH}"), refers to the value of the named environment variable.