GSP
Quick Navigator

Search Site

Unix VPS
A - Starter
B - Basic
C - Preferred
D - Commercial
MPS - Dedicated
Previous VPSs
* Sign Up! *

Support
Contact Us
Online Help
Handbooks
Domain Status
Man Pages

FAQ
Virtual Servers
Pricing
Billing
Technical

Network
Facilities
Connectivity
Topology Map

Miscellaneous
Server Agreement
Year 2038
Credits
 

USA Flag

 

 

Man Pages


Manual Reference Pages  -  RWSETTOOL (1)

.ds Aq ’

NAME

rwsettool - Operate on IPset files to produce a new IPset

CONTENTS

SYNOPSIS



  rwsettool { --union | --intersect | --difference
              | --mask=NET_BLOCK_SIZE | --fill-blocks=NET_BLOCK_SIZE
              | --sample {--size=SIZE | --ratio=RATIO} [--seed=SEED] }
        [--output-path=OUTPUT_PATH] [--record-version=VERSION]
        [--invocation-strip]
        [--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD] [INPUT_SET ...]

  rwsettool --help

  rwsettool --version



DESCRIPTION

rwsettool performs a single operation on one or more IPset file(s) to produce a new IPset file. The operations that rwsettool provides are union, intersection, difference, masking, and sampling. Details are provided in the OPTIONS section.

rwsettool reads the IPsets specified on the command line; when no IPsets are listed, rwsettool attempts to read an IPset from the standard input. The strings stdin or - can be used as the name of an input file to force rwsettool to read from the standard input. The resulting IPset is written to the location specified by the --output-path switch or to the standard output if that switch is not provided. Using the strings stdout or - as the argument to --output-path causes rwsettool to write the IPset to the standard output. rwsettool will exit with an error if an attempt is made to read an IPset from the terminal or write an IPset to the terminal.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

    Operation Switches

At least one of the following operation switches must be provided:
--union Perform the set union operation: The resulting IPset will contain an IP if that IP was present in any of the input IPsets.
--intersect Perform the set intersection operation: The resulting IPset will contain an IP if that IP was present in all of the input IPsets.
--difference Perform the set difference (relative complement) operation: The resulting IPset will contain an IP if that IP was present in first IPset and not present in any of the subsequent IPsets.
--mask=NET_BLOCK_SIZE Perform a (sparse) masking operation: The least significant 32-NET_BLOCK_SIZE or 128-NET_BLOCK_SIZE bits of each IP in every input IPset is set to zero, and the resulting IPset contains the union of these IPs. That is, the result contains one IP for each CIDR block of size NET_BLOCK_SIZE. NET_BLOCK_SIZE should be value between 1 and 32 for IPv4 sets, and between 1 and 128 for IPv6 sets. Contrast with --fill-blocks.
--fill-blocks=NET_BLOCK_SIZE Perform a masking operation that produces completely full blocks: The least significant 32-NET_BLOCK_SIZE or 128-NET_BLOCK_SIZE bits of each IP in every input IPset is set to zero. To create the output, each IP is modified to be a completely full NET_BLOCK_SIZE CIDR block.
--sample Select a random sample of IPs from the input IPsets. The size of the subset must be specified by either the --size or --ratio switches described below. In the case of multiple input IPsets, the resulting IPset is the union of all IP addresses sampled from each of the input IPsets.

    Sampling Switches

These switches control how records are sampled by the --sample operation.
--size=SIZE Select a random sample containing SIZE randomly selected records from each input IPset. If the input set is smaller than SIZE, all input IPs will be selected from that IPset.
--ratio=RATIO Select a random sample where the selection probability for each record of each input set is RATIO, specified as a decimal number between 0.0 and 1.0. The exact size of the subset selected from each file will vary between different runs with the same data.
--seed=SEED Seed the pseudo-random number generator with value SEED. By default, the seed will vary between runs. Seeding with specific values will produce repeatable results given the same input sets.

    Output Switches

These switches control the output:
--output-path=OUTPUT_PATH Write the resulting IPset to OUTPUT_PATH. If this switch is not provided, rwsettool will attempt to write the IPset to the standard output, unless it is connected to a terminal.
--record-version=VERSION Specify the format of the IPset records that are written to the output. Valid values are 0, 2, 3, and 4. When the switch is not provided, the SILK_IPSET_RECORD_VERSION environment variable is checked for a version. A VERSION of 2 creates a file compatible with SiLK 2.x, and it can only be used for IPsets containing IPv4 addresses. A VERSION of 3 creates a file that can only be read by SiLK 3.0 or later. A VERSION of 4 creates a file that can only be read by SiLK 3.7 or later. Version 4 files are smaller than version 3 files. The default VERSION is 0, which uses version 2 for IPv4 IPsets and version 3 for IPv6 IPsets.
--invocation-strip Do not record any command line history; that is, do not copy the invocation history from the input files to the output file, and do not record the current command line invocation in the output.
--note-strip Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.
--note-add=TEXT Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.
--note-file-add=FILENAME Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.
--compression-method=COMP_METHOD Specify how to compress the output. When this switch is not given, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.
none Do not compress the output using an external library.
zlib Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed.
lzo1x Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.
best Use lzo1x if available, otherwise use zlib. Only compress the output when writing to a file.
--help Print the available options and exit.
--version Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line.

Assume the following IPsets:



 A.set = { 1, 2, 4, 6 }
 B.set = { 1, 3, 5, 7 }
 C.set = { 1, 3, 6, 8 }
 D.set = { } (empty set)



Then the following commands will produce the following result IPsets:



 +---------------------------------+----------------------------+
 | OPTIONS                         | RESULT                     |
 +---------------------------------+----------------------------+
 | --union A.set B.set             | { 1, 2, 3, 4, 5, 6, 7 }    |
 | --union A.set C.set             | { 1, 2, 3, 4, 6, 8 }       |
 | --union A.set B.set C.set       | { 1, 2, 3, 4, 5, 6, 7, 8 } |
 | --union C.set D.set             | { 1, 3, 6, 8 }             |
 | --intersect A.set B.set         | { 1 }                      |
 | --intersect A.set C.set         | { 1, 6 }                   |
 | --intersect A.set B.set C.set   | { 1 }                      |
 | --intersect A.set D.set         | { }                        |
 | --difference A.set B.set        | { 2, 4, 6 }                |
 | --difference B.set A.set        | { 3, 5, 7 }                |
 | --difference A.set B.set C.set  | { 2, 4 }                   |
 | --difference C.set B.set A.set  | { 8 }                      |
 | --difference C.set D.set        | { 1, 3, 6, 8 }             |
 | --difference D.set C.set        | { }                        |
 +---------------------------------+----------------------------+



Sampling yields variable results, but here some example runs:



 +---------------------------------+----------------------------+
 | COMMAND                         | RESULT                     |
 +---------------------------------+----------------------------+
 | --sample -size 2 A.set          | { 1, 4 }                   |
 | --sample -size 2 A.set          | { 1, 6 }                   |
 | --sample -size 3 A.set          | { 2, 4, 6 }                |
 | --sample -size 2 A.set B.set    | { 1, 2, 5, 7 }             |
 | --sample -size 2 A.set B.set    | { 3, 4, 5, 6 }             |
 | --sample -size 2 A.set B.set    | { 1, 4, 5 }                |
 | --sample -ratio 0.5 A.set       | { 2, 6 }                   |
 | --sample -ratio 0.5 A.set       | { 4 }                      |
 | --sample -ratio 0.5 A.set B.set | { 1 }                      |
 | --sample -ratio 0.5 A.set B.set | { 2, 3, 5, 6, 7 }          |
 +---------------------------------+----------------------------+



These examples demonstrate some important points about sampling from IPsets:
o When using --size, an exact number of items is selected from each input set.
o When using --size with multiple input sets, the number of records in the output set may not be (num_input_sets*size) in all cases.
o When using --ratio, the number of items sampled is not stable between runs.
Given an IPset containing the three IPs



 10.1.1.1
 10.1.1.2
 10.1.3.1



specifying --mask=24 will produce an IPset containing two IPs:



 10.1.1.0
 10.1.3.0



while specifying --fill-blocks=24 will produce an IPset containing 512 IPs:



 10.1.1.0/24
 10.1.3.0/24



Suppose the IPset file mixed.set contains IPv4 and IPv6 addresses. To create an IPset file that contains only the IPv4 addresses, intersect mixed.set with an IPset that contains ::ffff:0:0/96.



 $ echo ::ffff:0:0/96 | rwsetbuild - all-v4.set

 $ rwsettool --intersect mixed.set all-v4.set > subset-v4.set



To create an IPset file that contains only the IPv6 addresses, subtract an IPset that contains ::ffff:0:0 from mixed.set:



 $ rwsettool --difference mixed.set all-v4.set > subset-v6.set



ENVIRONMENT

SILK_IPSET_RECORD_VERSION This environment variable is used as the value for the --record-version when that switch is not provided.
SILK_CLOBBER The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SEE ALSO

rwset(1), rwsetbuild(1), rwsetcat(1), rwfileinfo(1), silk(7), zlib(3)

NOTES

The --record-version switch was added in SiLK 3.0. Prior to SiLK 3.6, the only supported arguments for the switch were 2 and 3, with the default being 3. As of SiLK 3.6, the default is 0. Version 4 was added in SiLK 3.7.
Search for    or go to Top of page |  Section 1 |  Main Index


SiLK 3.11.0.1 RWSETTOOL (1) 2016-04-05

Powered by GSP Visit the GSP FreeBSD Man Page Interface.
Output converted with manServer 1.07.