NAME

rwpmapbuild - Create a binary prefix map from a text file

SYNOPSIS

  rwpmapbuild [--input-path=PATH] [--output-path=PATH]
        [--mode={ipv4|ipv6|proto-port}] [--dry-run] [--ignore-errors]
        [--note-add=TEXT] [--note-file-add=FILENAME]
        [--invocation-strip]

  rwpmapbuild --help

  rwpmapbuild --version

DESCRIPTION

rwpmapbuild reads a white-space-delimited stream of text and writes a binary output stream representing a prefix map. The syntax of this input is described in the "INPUT FILE FORMAT" section below.

The textual input is read from the file specified by --input-path or from the standard input when the switch is not provided. The binary output is written to the location named by --output-path or to the standard output when the switch is not provided and the standard output is not connected to a terminal.

A prefix map file is a binary file that maps a value (specifically either an IP addresses or a protocol-port pair) to a string label.

Once you have created a prefix map file, you may use the file in rwfilter(1), rwstats(1), rwuniq(1), rwgroup(1), rwsort(1), or rwcut(1) to partition, count, sort and display SiLK flow records based on the string labels defined in the prefix map. See the pmapfilter(3) manual page for details. To view the contents of a prefix map file, use rwpmapcat(1). To query the contents of a prefix map, use rwpmaplookup(1).

The remainder of this section provides example files that could be used as input to rwpmapbuild, and a note on the proper ordering of the input. For details on the syntax of the input, see the "INPUT FILE FORMAT" section that follows the description of the command line "OPTIONS".

Sample IPv4 input file

The following is a sample input file for rwpmapbuild that describes the registry of special-purpose IPv4 addresses. Any IP address that is not a special-purpose address get the label "Normal".

 # Prefix map sample input file for special purpose IPv4 addresses
 map-name   ipv4-special
 mode       ipv4
 default    Normal

 #    Each line has an either a CIDR block or a pair of IP
 #    addresses and then a label for that range
 0.0.0.0/8          This host on this network [RFC1122 section 3.2.1.3]
 10.0.0.0/8         Private-Use [RFC1918]
 100.64.0.0/10      Shared Address Space [RFC6598]
 127.0.0.0/8        Loopback [RFC1122 section 3.2.1.3]
 169.254.0.0/16     Link Local [RFC3927]
 172.16.0.0/12      Private-Use [RFC1918]
 192.0.0.0/24       IETF Protocol Assignments [RFC6890 section 2.1]
 192.0.0.0/29       IPv4 Service Continuity Prefix [RFC7335]
 #    A range of a single IP address requires a "/32" suffix or
 #    that the IP address be repeated
 192.0.0.8/32       IPv4 dummy address [RFC7600]
 192.0.0.9/32       Port Control Protocol Anycast [RFC7723]
 192.0.0.10/32      Traversal Using Relays around NAT Anycast [draft]
 #    A range may be specified as two IP addresses
 192.0.0.170  192.0.0.171   NAT64/DNS64 Discovery [RFC7050 section 2.2]
 192.0.2.0/24       Documentation (TEST-NET-1) [RFC5737]
 192.31.196.0/24    AS112-v4 [RFC7535]
 192.52.193.0/24    AMT [RFC7450]
 192.88.99.0/24     Deprecated (6to4 Relay Anycast) [RFC7526]
 192.168.0.0/16     Private-Use [RFC1918]
 192.175.48.0/24    Direct Delegation AS112 Service [RFC7534]
 198.18.0.0/15      Benchmarking [RFC2544]
 198.51.100.0/24    Documentation (TEST-NET-2) [RFC5737]
 203.0.113.0/24     Documentation (TEST-NET-3) [RFC5737]
 240.0.0.0/4        Reserved [RFC1112 section 4]
 255.255.255.255/32 Limited Broadcast [RFC919 section 7]

Sample IPv6 input file

The following input file for rwpmapbuild describes IPv6 address space. The file demonstrates the use of the label statement. It uses a hyphen ("-") as the label for any undefined ranges.

 # Prefix map sample input file for IPv6 address space
 map-name   iana-ipv6
 mode       ipv6

 label      0   RFC3849
 label      1   RFC3879
 label      2   RFC4048
 label      3   RFC4193
 label      4   RFC4291
 label      5   RFC4291 Loopback Address
 label      6   RFC4291 Unspecified Address
 label      7   RFC4291 IPv4-mapped Address
 label      8   RFC5180
 label      9   RFC6666
 label     10   RFC7723
 label     11   -

 default        -

 0000::/8       4
 ::1/128        5
 ::/128         6
 ::ffff:0:0/96  7
 0100::/8       4
 0100::/64      9           # RFC6666
 0200::/7       2           # RFC4048
 0400::/6       4
 0800::/5       4
 1000::/4       4
 2000::/3       4
 2001:1::1/128 10           # RFC7723
 2001:2::/48    8           # Benchmarking
 2001:db8::/32  0           # Documentation
 4000::/3       4
 6000::/3       4
 8000::/3       4
 a000::/3       4
 c000::/3       4
 e000::/4       4           # You may use the label number or the
 f000::/5       RFC4291     # exact label name, but any other text
 f800::/6       4           # causes rwpmapbuild to issue an error
 fc00::/7       RFC4193
 fe00::/9       4
 fe80::/10      4
 fec0::/10      RFC3879
 ff00::/8       4

Sample protocol-port input file

This is a small sample of a file that could be used to label IP protocols, specific ports within the TCP and UDP protocols, and ICMP type and code values. When ranges overlap or one range is a specialization of another, the wider or more general range should be listed first, followed by the narrower or more specific ranges.

 map-name   protocol-port-example
 mode       proto-port

 #    The range is either a single protocol or a protocol and
 #    a port separated by a slash.
 1      1           ICMP
 #    Specify the wider categories first, then specialize
 6      6           TCP
 6/0    6/1024      TCP/Generic reserved
 #    A range of a single port requires both the starting
 #    value and the ending value
 6/21   6/21        TCP/FTP
 6/22   6/22        TCP/SSH
 6/25   6/25        TCP/SMTP
 6/80   6/80        TCP/HTTP
 6/443  6/443       TCP/HTTPS
 6/6000 6/6063      TCP/X11
 17     17          UDP
 17/0   17/1024     UDP/Generic reserved
 17/53  17/53       UDP/DNS
 17/67  17/68       UDP/DHCP
 50     50          ESP
 58     58          ICMPv6
 #    For ICMP Type/Code, convert the type and code to a port
 #    value using this expression:  type * 256 + code
 1/0    1/255       ICMP/Echo Reply
 1/768  1/1023      ICMP/Destination Unreachable
 1/1024 1/1279      ICMP/Source Quench
 1/768  1/768       ICMP/Net Unreachable
 1/769  1/769       ICMP/Host Unreachable
 1/770  1/770       ICMP/Protocol Unreachable
 1/771  1/771       ICMP/Port Unreachable

Complete ICMPv4 Prefix Map

An ideal candidate for port-based prefix maps is for decoding ICMP types and codes. Although most SiLK commands support a form of ICMP type and code options, these are all based on the actual number values. However, a prefix map may be useful to decode the noun-name of the ICMP types and codes. The following prefix map can be used for that purpose. (Note that ICMP type and code is always in the destination port field, regardless of the traffic direction.)

 # Identify this as a protocol-port prefix map, rather than an IP-range map
 mode proto-port

 # Set a default value for all records
 0 255 Other

 # Set the default value for all ICMP records
 1 1 ICMP/Undefined

 # ICMP specific entries
 1/0 1/255 ICMP/Echo Reply
 1/768 1/768 ICMP/Destination Unreachable/Net Unreachable
 1/769 1/769 ICMP/Destination Unreachable/Host Unreachable
 1/770 1/770 ICMP/Destination Unreachable/Protocol Unreachable
 1/771 1/771 ICMP/Destination Unreachable/Port Unreachable
 1/772 1/772 ICMP/Destination Unreachable/Fragmentation Needed and Don't Fragment was Set
 1/773 1/773 ICMP/Destination Unreachable/Source Route Failed
 1/774 1/774 ICMP/Destination Unreachable/Destination Network Unknown
 1/775 1/775 ICMP/Destination Unreachable/Destination Host Unknown
 1/776 1/776 ICMP/Destination Unreachable/Source Host Isolated
 1/777 1/777 ICMP/Destination Unreachable/Communication with Destination Network is Administratively Prohibited
 1/778 1/778 ICMP/Administratively Prohibited/Communication with Destination Host is Administratively Prohibited
 1/779 1/779 ICMP/Administratively Prohibited/Destination Network Unreachable for Type of Service
 1/780 1/780 ICMP/Administratively Prohibited/Destination Host Unreachable for Type of Service
 1/781 1/781 ICMP/Administratively Prohibited/Communication Administratively Prohibited
 1/782 1/782 ICMP/Administratively Prohibited/Host Precedence Violation
 1/783 1/783 ICMP/Administratively Prohibited/Precedence cutoff in effect
 1/1024 1/1279 ICMP/Source Quench
 1/1280 1/1280 ICMP/Redirect/Redirect Datagram for the Network (or subnet)
 1/1281 1/1281 ICMP/Redirect/Redirect Datagram for the Host
 1/1282 1/1282 ICMP/Redirect/Redirect Datagram for the Type of Service and Network
 1/1283 1/1283 ICMP/Redirect/Redirect Datagram for the Type of Service and Host
 1/1536 1/1536 ICMP/Alternate Host Address/Alternate Address for Host
 1/2048 1/2303 ICMP/Echo
 1/2304 1/2304 ICMP/Router Advertisement/Normal router advertisement
 1/2320 1/2320 ICMP/Router Advertisement/Does not route common traffic
 1/2560 1/2815 ICMP/Router Selection
 1/2816 1/2816 ICMP/Time Exceeded/Time to Live exceeded in Transit
 1/2817 1/2817 ICMP/Time Exceeded/Fragment Reassembly Time Exceeded
 1/3072 1/3072 ICMP/Parameter Problem/Pointer indicates the error
 1/3073 1/3073 ICMP/Parameter Problem/Missing a Required Option
 1/3074 1/3074 ICMP/Parameter Problem/Bad Length
 1/3328 1/3583 ICMP/Timestamp
 1/3584 1/3839 ICMP/Timestamp Reply
 1/3840 1/4095 ICMP/Information Request
 1/4096 1/4351 ICMP/Information Reply
 1/4352 1/4607 ICMP/Address Mask Request
 1/4608 1/4863 ICMP/Address Mask Reply
 1/7680 1/7935 ICMP/Traceroute
 1/7936 1/8191 ICMP/Datagram Conversion Error
 1/8192 1/8447 ICMP/Mobile Host Redirect
 1/8448 1/8703 ICMP/IPv6 Where-Are-You
 1/8704 1/8959 ICMP/IPv6 I-Am-Here
 1/8960 1/9215 ICMP/Mobile Registration Request
 1/9216 1/9471 ICMP/Mobile Registration Reply
 1/9984 1/10239 ICMP/SKIP
 1/10240 1/10240 ICMP/Photuris/Bad SPI
 1/10241 1/10241 ICMP/Photuris/Authentication Failed
 1/10242 1/10242 ICMP/Photuris/Decompression Failed
 1/10243 1/10243 ICMP/Photuris/Decryption Failed
 1/10244 1/10244 ICMP/Photuris/Need Authentication
 1/10245 1/10245 ICMP/Photuris/Need Authorization

 # A few other well-known protocols
 6 6 TCP
 17 17 UDP
 50 50 ESP
 51 51 AH

Proper Ordering of rwpmapbuild Input

When creating the textual input for rwmpabuild, be sure to put the most general attributes first in the list.

For example, suppose we administer the address block 12.0.0.0/8, and would like to report on address ranges delegated within the organization, A prefix map can be used as follows to show utilization for each address block, as well as unallocated (and presumably unauthorized) usage.

Display the contents of input file:

 $ cat network.pmap.txt
 12.0.0.0/8 Assigned, Unallocated
 12.1.0.0/16 RESERVED
 12.38.0.0/16 Client Network 1
 12.127.0.0/16 Data Center (Primary)
 12.130.0.0/16 Client Network 2
 12.154.0.0/16 Client Network 3
 12.186.0.0/16 Data Center (Secondary)
 12.210.0.0/16 RESERVED

Create the binary prefix map:

 $ rwpmapbuild --input=network.pmap.txt --output=network.pmap

Use rwfilter(1) to select IPs in the 12.0.0.0/8 netblock, and use rwuniq(1) to bin the results according to the prefix map:

 $ rwfilter --start=2007/07/30:00 --saddr=12.x.x.x --pass=stdout  \
   | rwuniq --pmap-file=network.pmap --field=sval --value=bytes

                    sval|               Bytes|
                RESERVED|               39749|
   Data Center (Primary)|               87621|
   Assigned, Unallocated|             4296212|
        Client Network 2|              545848|
 Data Center (Secondary)|               18228|
        Client Network 1|              112404|
        Client Network 3|               68820|

Suppose the input file had placed the most general entry at the bottom, like so:

 $ cat network.pmap.txt
 12.1.0.0/16 RESERVED
 12.38.0.0/16 Client Network 1
 12.127.0.0/16 Data Center (Primary)
 12.130.0.0/16 Client Network 2
 12.154.0.0/16 Client Network 3
 12.186.0.0/16 Data Center (Secondary)
 12.210.0.0/16 RESERVED
 12.0.0.0/8 Assigned, Unallocated

The general entry would overwrite the other entries:

 $ rwpmapbuild --input=network.pmap.txt --output=network.pmap
 $ rwfilter --start=2007/07/30:00 --saddr=12.x.x.x --pass=stdout \
   | rwuniq --pmap-file=network.pmap --fields=sval --bytes
                    sval|               Bytes|
   Assigned, Unallocated|             5168882|

This is easy to see by using rwpmapcat(1) to print the contents of the prefix map:

 $ rwpmapcat --map-file=network.pmap
          ipBlock|                  label|
        0.0.0.0/5|                UNKNOWN|
        8.0.0.0/6|                UNKNOWN|
       12.0.0.0/8|  Assigned, Unallocated|
       13.0.0.0/8|                UNKNOWN|
       14.0.0.0/7|                UNKNOWN|
       16.0.0.0/4|                UNKNOWN|
       32.0.0.0/3|                UNKNOWN|
       64.0.0.0/2|                UNKNOWN|
      128.0.0.0/1|                UNKNOWN|

(rwpmapcat lists all possible addresses from 0.0.0.0 to 255.255.255.255 and their labels. The default label is "UNKNOWN" unless the default is set to something else.)

The best way to make sure your entries are properly ordered is to explicitly order them before compiling the prefix map. When the data uses the CIDR-block format, the UNIX sort(1) command often produces the proper output.

 $ cat network.pmap.txt
 12.1.0.0/16 RESERVED
 12.38.0.0/16 Client Network 1
 12.127.0.0/16 Data Center (Primary)
 12.130.0.0/16 Client Network 2
 12.154.0.0/16 Client Network 3
 12.186.0.0/16 Data Center (Secondary)
 12.210.0.0/16 RESERVED
 12.0.0.0/8 Assigned, Unallocated

Split the input at the "/" and sort the input numerically by the bitmask size. (A small bitmask represents a large netblock.)

 $ sort -n -k 2 -t "/" network.pmap.txt
 12.0.0.0/8 Assigned, Unallocated
 12.1.0.0/16 RESERVED
 12.127.0.0/16 Data Center (Primary)
 12.130.0.0/16 Client Network 2
 12.154.0.0/16 Client Network 3
 12.186.0.0/16 Data Center (Secondary)
 12.210.0.0/16 RESERVED
 12.38.0.0/16 Client Network 1

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--input-path=PATH: Read the textual input from PATH. You may use "stdin" or "-" to represent the standard input. When this switch is not provided, the input is read from the standard input unless the standard input is a terminal. rwpmapbuild will read textual input from the terminal if the standard input is explicitly specified as the input. The input file format is described below. (Added in SiLK 3.17.0 as a replacement for --input-file.)
--output-path=PATH: Write the binary prefix map to PATH. You may use "stdout" or "-" to represent the standard output. When this switch is not provided, the prefix map is written to the standard output unless the standard output is connected to a terminal. (Added in SiLK 3.17.0 as a replacement for --output-file.)
--mode={ipv4|ipv6|proto-port}: Specify the type of the input, as if a mode statement appeared in the input stream. The value specified by this switch must not conflict with an explicit mode statement appearing in the input.
--dry-run: Do not write the output file. Simply check the syntax of the input file.
--ignore-errors: Write the output file regardless of any errors encountered while parsing the input file.
--note-add=TEXT: Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.
--note-file-add=FILENAME: Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.
--invocation-strip: Do not record the command used to create the prefix map in the output. When this switch is not given, the invocation is written to the file's header, and the invocation may be viewed with rwfileinfo(1). Since SiLK 3.12.0.
--input-file=PATH: Read the textual input from PATH. An alias for --input-path. Deprecaated as of SiLK 3.17.0.
--output-file=PATH: Write the binary prefix map to PATH. An alias for --output-path. Deprecaated as of SiLK 3.17.0.
--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

INPUT FILE FORMAT

This section describes the format of the textual input file for rwpmapbuild. Three example files are shown above in the "DESCRIPTION" section.

Blank lines or lines containing only whitespace in the input file are ignored.

The file may contain comments, and these are ignored. A comment begins with the first "#" character on a line and extends to the end of the line. Note that "#" appearing in a textual label is treated as the beginning of a comment.

Each non-blank line in the input file that is not a comment is considered a statement. A statement must be completed on a single line, and only one statement may appear on a line.

The delimiter in the input file is whitespace---specifically one or more space and/or tab characters.

rwpmapbuild supports five types of statements. Four of those statements begin with a specific keyword: one of mode, map-name, label, and default. Any line that does not begin with with a keyword is expected to contain a range definition, which maps a range to a label. The format of the range definition depends on the mode.

The four statement types that begin with a keyword are all optional. They are:

mode { ipv4 | ipv6 | proto-port | ip }

Specify what types of ranges are defined in the file. The mode statement must appear before any ranges are specified. The mode may also be set using the --mode command line switch. When both the mode statement and the --mode switch are given, their values must match. When neither the mode statement nor the --mode switch is provided, rwpmapbuild processes the input in IPv4 address mode. The ip mode is deprecated; it is an alias for ipv4. The mode statement may only appear one time.

map-name simple-string

Create a name for the data in this prefix map file that other SiLK tools may use to refer to this prefix map file. When the prefix map file is used by rwfilter(1), the simple-string is used to generate the filtering switch names. When the prefix map file is used by rwcut(1), rwgroup(1), rwsort(1), rwstats(1), or rwuniq(1), the simple-string is used to generate the field names. See pmapfilter(3) for details. The simple-string may not contain whitespace, a comma, or a colon. The map-name statement may only appear one time.

label num label-text

Associate the numeric identifier num with the given label text label-text. By specifying a label statement, the identifier num is expanded to label-text when the range definitions are being defined.

Either all labels used in the file must appear in label statements, or no label statements may appear in which case rwpmapbuild creates labels as it parses the range definitions. All label statements must appear before the default statement and before the range definitions.

label-text is a textual string that begins at the first non-whitespace character and extends to the final non-whitespace character on that line that does not appear in a comment. The label-text may include embedded whitespace and non-alphanumeric characters. While a comma (",") is legal in the label-text, using a comma prevents the label from being used by the --pmap-src and --pmap-dest switches in rwfilter(1).

If no label statements appear in the input, any text containing at least one non-whitespace character may be used as the label in the default statement and the range definitions.

It is an error if num or label-text appear in any other label statement. The minimum value for num is 0 and the maximum value is 2147483647. Note that rwpmapbuild creates labels for all numeric identifiers between 0 and the maximum identifier used in the input file, and using an unnecessarily large value creates many empty entries.

default label-value

Use the label label-value for any ranges not explicitly mentioned in this input file. The label-value text is one of

1.: when label statements are used, a numerical label identifier that was specified in one of the statements
2.: when label statements are used, a string that is an exact match of the label-text that was specified in one of those statements
3.: when label statements are not used, a string that begins at the first non-whitespace character and extends to the final non-whitespace character on that line that does not appear in a comment

The default statement must appear before the range definitions are specified. If the default statement does not appear in the input, the label "UNKNOWN" is automatically defined and used as the default.

As mentioned above, any line that does not begin with one of the above keywords must contain a range definition, and the format of the line depends on the type of data in the input file--that is, the mode of the input.

Regardless of the input mode, the final item in each range definition is the label to assign to that range. The label has the same form as that described for the default statement above, and the label is denoted by label-value in the following.

Address Mode

When the mode is either ipv4 or ipv6, rwpmapbuild parses the file in address mode. In address mode, each range definition contains an IP range and a label-value.

A range is either a CIDR block or a pair of IP addresses that specify the starting IP and ending IP of the range. To provide a label for a single IP address, you must either use the same IP address as the starting and ending values of the range, append "/32" to a single IPv4 address, or append "/128" to a single IPv6 address. When the mode is ipv4, an IPv6 address in the input file raises an error. rwpmapbuild also accepts integer representations of IP addresses when in ipv4 mode,

When ranges overlap or one range is a specialization of another, the wider or more general range should be listed first, followed by the narrower or more specific ranges.

cidr-block label-value: Associate the given label identifier or label text with this CIDR block. The CIDR block is composed of an IP address in canonical notation (e.g, dotted-decimal for IPv4), a slash "/", and the number of significant bits.
low-ip high-ip label-value: Associate the given label identifier or label text with this IP range, where low-ip and high-ip are in canonical notation.
low-int high-int label-value: Treat low-int and high-int as 32-bit values, convert the values to IPv4 addresses, and associate the given label identifier or label text with the IPv4 range.

Protocol/Port Mode

When the mode is proto-port, rwpmapbuild parses the file in protocol/port mode. In protocol/port mode, each range definition contains a starting value, an ending value, and the label-value.

The starting and ending values may both be integers between 0 and 255, inclusive. The numbers are treated as protocol values, where 6 is TCP, 17 is UDP, and 1 is ICMP.

The starting and ending values may also both be a number, a slash ("/"), and a number. The first number is treated as the protocol and the second number as a port number (or service) in that protocol. For example, 6/80 is considered the http service of TCP.

When ranges overlap or one range is a specialization of another, the wider or more general range should be listed first, followed by the narrower or more specific ranges. That is, specify the generic protocol first, then port numbers within that protocol.

proto/port proto/port label-value: Associate the given label identifier or label text with all protocols and port numbers between these two values inclusive. Note that while port is not meaningful for all protocols (specifically, it is meaningful for TCP and UDP and may contain type/code information for ICMP), rwpmapbuild accepts port numbers for any protocol.
proto proto label-value: Associate the given label identifier or label text for all protocols between these two values inclusive.

EXAMPLE

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

Reading from and writing to a file:

 $ rwpmapbuild --input iana-ipv6.txt --output iana-ipv6.pmap

Reading from the standard input and writing to the standard output:

 $ cat ipv4-special.txt              \
   | rwpmapbuild > ipv4-special.pmap

For example input files, see the "DESCRIPTION" section above.

ENVIRONMENT

SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.