Dotted decimal---all 4 octets are required:
An unsigned 32-bit integer:
An IPv6 address in canonical form (when SiLK has been compiled with
Any of the above with a CIDR designation---for dotted decimal all
four octets are still required:
SiLK IP wildcard notation. A SiLK IP Wildcard can represent multiple
IPv4 or IPv6 addresses. An IP Wildcard contains an IP in its
canonical form, except each part of the IP (where part is an octet
for IPv4 or a hexadectet for IPv6) may be a single value, a range, a
comma separated list of values and ranges, or the letter x to
signify all values for that part of the IP (that is, 0-255 for
IPv4). You may not specify a CIDR suffix when using the IP Wildcard
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.
The following two switches control the type of input; one and only one must be provided:
--set-input=SETFILE Create a Bag from an IPset. SETFILE is a filename, a named pipe, or the keyword stdin or - to read the IPset from the standard input. Counts have a volume of 1 when the --default-count switch is not specified. (IPsets are typically created by rwset(1) or rwsetbuild(1).) --bag-input=TEXTFILE Create a Bag from a delimited text file. TEXTFILE is a filename, a named pipe, or the keyword stdin or - to read the text from the standard input. See the DESCRIPTION section for the syntax of the TEXTFILE. --delimiter=C The delimiter to expect between each key-count pair of the TEXTFILE read by the --bag-input switch. The default delimiter is the vertical pipe (|). The delimiter is ignored if the --set-input switch is specified. When the delimiter is a whitespace character, any amount of whitespace may surround and separate the key and counter. Since # is used to denote comments and newline is used to denote records, neither is a valid delimiter character. --default-count=DEFAULTCOUNT Override the counts of all values in the input text or IPset with the value of DEFAULTCOUNT. DEFAULTCOUNT must be a positive integer. --key-type=FIELD_TYPE Write a entry into the header of the Bag file that specifies the key contains FIELD_TYPE values. When this switch is not specified, the key type of the Bag is set to custom. The FIELD_TYPE is case insensitive. The supported FIELD_TYPEs are:
sIPv4 source IP address, IPv4 only dIPv4 destination IP address, IPv4 only sPort source port dPort destination port protocol IP protocol packets packets, see also sum-packets bytes bytes, see also sum-bytes flags bitwise OR of TCP flags sTime starting time of the flow record, seconds resolution duration duration of the flow record, seconds resolution eTime ending time of the flow record, seconds resolution sensor sensor ID input SNMP input output SNMP output nhIPv4 next hop IP address, IPv4 only initialFlags TCP flags on first packet in the flow sessionFlags bitwise OR of TCP flags on all packets in the flow except the first attributes flow attributes set by the flow generator application guess as to the content of the flow, as set by the flow generator class class of the sensor type type of the sensor icmpTypeCode an encoded version of the ICMP type and code, where the type is in the upper byte and the code is in the lower byte sIPv6 source IP, IPv6 dIPv6 destination IP, IPv6 nhIPv6 next hop IP, IPv6 records count of flows sum-packets sum of packet counts sum-bytes sum of byte counts sum-duration sum of duration values any-IPv4 a generic IPv4 address any-IPv6 a generic IPv6 address any-port a generic port any-snmp a generic SNMP value any-time a generic time value, in seconds resolution custom a number --counter-type=FIELD_TYPE Write a entry into the header of the Bag file that specifies the counter contains FIELD_TYPE values. When this switch is not specified, the counter type of the Bag is set to custom. The supported FIELD_TYPEs are the same as those for the key. --note-add=TEXT Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool. --note-file-add=FILENAME Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation. --compression-method=COMP_METHOD Specify how to compress the output. When this switch is not given, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.
none Do not compress the output using an external library. zlib Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed. lzo1x Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. best Use lzo1x if available, otherwise use zlib. Only compress the output when writing to a file. --output-path=OUTPUTFILE Redirect output to OUTPUTFILE. OUTPUTFILE is a filename, a named pipe, or the keyword stdout or - to write the bag to the standard output. --help Print the available options and exit. --version Print the version number and information about how SiLK was configured, then exit the application.
In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.
Assume the file mybag.txt contains the following lines, where each line contains an IP address, a comma as a delimiter, a count, and ends with a newline.
192.168.0.1,5 192.168.0.2,500 192.168.0.3,3 192.168.0.4,14 192.168.0.5,5
To build a bag with it:
$ rwbagbuild --bag-input=mybag.txt --delimiter=, > mybag.bag
Use rwbagcat(1) to view its contents:
$ rwbagcat mybag.bag 192.168.0.1| 5| 192.168.0.2| 500| 192.168.0.3| 3| 192.168.0.4| 14| 192.168.0.5| 5|
To create a Bag of protocol data from the text file myproto.txt:
1| 4| 6| 138| 17| 131|
$ rwbagbuild --key-type=proto --bag-input=myproto.txt > myproto.bag $ rwbagcat myproto.bag 1| 4| 6| 138| 17| 131|
When the --key-type switch is specified, rwbagcat knows the keys should be printed as integers, and rwfileinfo(1) shows the type of the key:
$ rwfileinfo --fields=bag myproto.bag myproto.bag: bag key: protocol @ 4 octets; counter: custom @ 8 octets
Without the --key-type switch, rwbagbuild assumes the integers in myproto.txt represent IP addresses:
$ rwbagbuild --bag-input=myproto.txt | rwbagcat 0.0.0.1| 4| 0.0.0.6| 138| 0.0.0.17| 131|
Although the --integer-keys switch on rwbagcat forces it to print keys as integers, it is generally better to use the --key-type switch when creating the bag.
$ rwbagbuild --bag-input=myproto.txt | rwbagcat --integer-keys
To ignore the counts that exist in myproto.txt and set the counts for each protocol to 1, use the --default-count switch which overrides the existing value:
$ rwbagbuild --key-type=protocol --bag-input=myproto.txt \ --default-count=1 --output-path=myproto1.bag $ rwbagcat myproto1.bag 1| 1| 6| 1| 17| 1|
Given the IP set myset.set, create a bag where every entry in the bag has a count of 3:
$ rwbagbuild --set-input=myset.set --default-count=3 \ --out=mybag2.bag
Suppose we have three IPset files, A.set, B.set, and C.set:
$ rwsetcat A.set 10.0.0.1 10.0.0.2 $ rwsetcat B.set 10.0.0.2 10.0.0.3 $ rwsetcat C.set 10.0.0.1 10.0.0.2 10.0.0.4
We want to create a bag file from these IPset files where the count for each IP address is the number of files that IP appears in. rwbagbuild accepts a single file as an argument, so we cannot do the following:
$ rwbagbuild --set-input=A.set --set-input=B.set ... # WRONG!
(Even if we could repeat the --set-input switch, specifying it multiple times would be annoying if we had 300 files instead of only 3.)
The IPset files are (mathematical) sets, so if we join them together first with rwsettool(1) and then run rwbagbuild, each IP address gets a count of 1:
$ rwsettool --union A.set B.set C.set \ | rwbagbuild --set-input=- \ | rwbagcat 10.0.0.1| 1| 10.0.0.2| 1| 10.0.0.3| 1| 10.0.0.4| 1|
When rwbagbuild is processing textual input, it sums the counters for keys that appear in the input multiple times. We can use rwsetcat(1) to convert each IPset file to text and feed that as single textual stream to rwbagbuild. Use the --cidr-blocks switch on rwsetcat to reduce the amount of input that rwbagbuild must process. This is probably the best approach to the problem:
$ rwsetcat --cidr-block *.set | rwbagbuild --bag-input=- > total1.bag $ rwbagcat total1.bag 10.0.0.1| 2| 10.0.0.2| 3| 10.0.0.3| 1| 10.0.0.4| 1|
A less efficient solution is to convert each IPset to a bag and then use rwbagtool(1) to add the bags together:
$ for i in *.set ; do rwbagbuild --set-input=$i --output-file=/tmp/$i.bag ; done $ rwbagtool --add /tmp/*.set.bag > total2.bag $ rm /tmp/*.set.bag
There is no need to create a bag file for each IPset; we can get by with only two bag files, the final bag file, total3.bag, and a temporary file, tmp.bag. We initialize total3.bag to an empty bag. As we loop over each IPset, rwbagbuild converts the IPset to a bag on its standard output, rwbagtool creates tmp.bag by adding its standard input to total3.bag, and we rename tmp.bag to total3.bag:
$ rwbagbuild --bag-input=/dev/null --output-file=total3.bag $ for i in *.set ; do rwbagbuild --set-input=$i \ | rwbagtool --output-file=tmp.bag --add total3.bag stdin ; /bin/mv tmp.bag total3.bag ; done $ rwbagcat total3.bag 10.0.0.1| 2| 10.0.0.2| 3| 10.0.0.3| 1| 10.0.0.4| 1|
SILK_CLOBBER The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.
rwbag(1), rwbagcat(1), rwbagtool(1), rwfileinfo(1), rwset(1), rwsetbuild(1), rwsetcat(1), rwsettool(1), silk(7), zlib(3)
The --default-count switch is poorly named.
|SiLK 22.214.171.124||RWBAGBUILD (1)||2016-04-05|