|
|
| |
sensor.conf(5) |
SiLK Tool Suite |
sensor.conf(5) |
sensor.conf - Sensor Configuration file for rwflowpack and flowcap
As part of collecting flow data, the rwflowpack(8) and
flowcap(8) daemons need to know what type of data they
are collecting and how to collect it (e.g., listen on 10000/udp for
NetFlow v5; listen on 4740/tcp for IPFIX). In addition, the
rwflowpack daemon needs information on how to categorize the flow: for
example, to label the flows collected at a border router as incoming or
outgoing. The Sensor Configuration file, sensor.conf, contains this
information, and this manual page describes the syntax of the file (see
"SYNTAX" below) and provides some example configurations (see
"EXAMPLES").
The sensor.conf file may have any name, and it may reside
in any location. The name and location of the file is specified by the
--sensor-configuration switch to rwflowpack and
flowcap.
The Sensor Configuration File defines the following concepts:
- probe
- A probe specifies a source for flow data. The source could be a port on
which flowcap or rwflowpack collects NetFlow or IPFIX data
from a flow generator such as a router or the yaf software
(<http://tools.netsa.cert.org/yaf/>). In rwflowpack, the
source can be a directory to periodically poll for files containing
NetFlow v5 PDUs, IPFIX records, or SiLK Flow records. When defining a
probe, you must specify a unique name for the probe and the probe's
type.
- group
- A group is a named list that contains one of the following: CIDR blocks,
the names of IPset files, or integers representing SNMP interfaces or VLAN
identifiers. The use of groups is optional; the primary purpose of a group
is to allow the administrator to specify a commonly used list (such as the
IP space of the network being monitored) in a single location.
- sensor
- A sensor represents a logical collection point for the purposes of
analysis. The sensor contains configuration values that rwflowpack
uses to categorize each flow record depending on how the record moves
between networks at the collection point. Since the sensors and the
categories (known as flowtypes or as class/type
pairs) are also used for analysis, they are defined in the Site
Configuration file, described in silk.conf(5). The
Sensor Configuration file maps sensors to probes and specifies the rules
required to categorize the data. Usually one sensor corresponds to one
probe; however, a sensor may be comprised of multiple probes, or the flow
data collected at a single probe may be handled by multiple sensors.
The next section of this manual page describes the syntax of the
sensor.conf file.
Using the syntax to configure a sensor requires knowledge of the
packing logic that rwflowpack is using. The packing logic is
the set of rules that rwflowpack uses to assign a flowtype to each
record it processes. The default packing logic is for the twoway
site, which is described in the packlogic-twoway(3)
manual page. Additional packing logic rules are available (e.g.,
packlogic-generic(3)).
The last major section of this document is "EXAMPLES"
where several common configurations are shown. These examples assume
rwflowpack is using the packing logic from the twoway
site.
When parsing the Sensor Configuration file, blank lines are ignored. At any
location in a line, the character "#"
indicates the beginning of a comment, which continues to the end of the line.
These comments are ignored.
All other lines begin with optional leading whitespace, a command
name, and one or more arguments to the command. Command names are a sequence
of non-whitespace characters, not including the character
"#". Arguments are textual atoms: any
sequence of non-whitespace, non-"#"
characters, including numerals and punctuation.
There are four contexts for commands: top-level, probe block,
group block, and sensor block. The probe block, group block, and sensor
block contexts are used to describe individual features of probes, groups,
and sensors, respectively.
The valid commands for each context are described below.
In addition to the commands to begin a probe, group, or sensor block, the
top-level context supports the following command:
- include "path"
- The include command is used to include the contents of another file
whose location is path. This may be used to separate large
configurations into logical units. The argument to include must be
a double-quoted string.
With the exception of the probe command, the commands listed below are
accepted within the probe context. Within a probe block, one and only one of
the following must be specified: listen-on-port to listen on a network
socket, poll-directory to poll a directory for files,
read-from-file to read a single file, or listen-on-unix-socket
to listen on a UNIX domain socket. These commands are described below.
- probe probe-name probe-type
- The probe command is used in the top-level context to begin a new
probe block which continues to the end probe command. The
arguments to the probe command are the name of the probe being
defined and the probe type. The probe-name must be unique among all
probes. It must begin with a letter, and it may not contain whitespace
characters or the slash character ("/").
When a probe is associated with a single sensor, it is good practice to
give the probe the same name as the sensor. The probe-type must be
one of the following:
- netflow-v5
- This probe processes NetFlow v5 protocol data units (PDU) that the daemon
reads from a UDP port or from a file. NetFlow may be generated by a router
or by software that reads packet capture (pcap(3))
data and generates NetFlow v5 records.
- netflow
- This is an alias for netflow-v5 for backwards compatibility. This
alias is deprecated, and it may be removed in a future release.
- ipfix
- An IPFIX probe processes Internal Protocol Flow Information eXchange
records that the daemon reads over the network from an IPFIX source such
as yaf(1). An IPFIX probe can also poll a directory
for files generated by the yaf program. To support IPFIX probes,
SiLK must be built with support for the external library libfixbuf,
version 1.7.0 or later. Both yaf and libfixbuf are available from
<http://tools.netsa.cert.org/>.
- netflow-v9
- This probe processes NetFlow v9 protocol data units (PDU) that the daemon
reads from a UDP port from a router. To support NetFlow v9 probes, SiLK
must be built with support for the external library libfixbuf, version
1.7.0 or later.
- sflow
- This probe processes sFlow v5 records that the daemon reads from a UDP
port. To support sFlow probes, SiLK must be built with support for the
external library libfixbuf, version 1.7.0 or later. Since SiLK
3.9.0.
- silk
- A SiLK probe processes the records contained in SiLK Flow files created by
previous invocations of rwflowpack. The flows will be completely
re-packed, as if they were just received over the network. The sensor and
flowtype values in each flow will be ignored. Note that SiLK usually
removes the SNMP interfaces from its flow records, and it is likely that
you will be unable to use the SNMP interfaces to pack the flows.
- end probe
- The end probe command ends the definition of a probe. Following an
end probe command, top-level commands are again accepted.
- listen-on-port port
- This command configures the probe to accept flow records over the network,
and port specifies the network port number where the probe should
listen for flow data. The protocol command is required when
listen-on-port is specified, and the listen-as-host and
accept-from-host commands are optional. Multiple probes may use the
same value for port as long as the probes are the same type and the
accept-from-host command is specified in each probe block. Probes
of different types may not bind to the same port, meaning the combination
of the following three values must be different: listen-on-port,
protocol, and listen-as-host. When listening to IPFIX data
from yaf, this is the value specified to yaf's
--ipfix-port switch. When listening to NetFlow from a Cisco router,
this is the port that was specified to the Cisco IOS command
ip flow-export [ip-address] [port]
- protocol { tcp | udp }
- This command, required when listen-on-port is given, specifies
whether the port is a "tcp" or
"udp" port. IPFIX probes support both
types; the only permitted value for all other probe types is
"udp". When listening to IPFIX data from
yaf, this is the value specified to yaf's --ipfix
switch.
- accept-from-host host-name [host-name...]
- This optional command specifies the hosts that are allowed to connect to
the port where the probe is listening. The argument is a list of IP
addresses and/or hostnames separated by whitespace and/or a comma. When
this command is not present, any host may connect. The command may only be
specified when the listen-on-port command is also present. When
multiple probes use the same listen-on-port, protocol, and
listen-as-host values, the accept-from-host switch must be
used so that rwflowpack may assign incoming records to a specified
probe. When listening for NetFlow, this attribute would be the IP address
of the router as seen from the machine running rwflowpack or
flowcap. (Prior to SiLK 3.10.1, the accept-from-host command
accepted only a single argument.)
- listen-as-host host-name
- This optional command is used on a multi-homed machine to specify the
address the probe should listen on (bind(2) to). Its value is the
name of the host or its IP address. If not present, the program will
listen on all the machine's addresses. The command may only be specified
when the listen-on-port command is also present. For listening to
NetFlow, the value would be the ip-address that was specified to
the Cisco IOS command
ip flow-export [ip-address] [port]
- listen-on-unix-socket path-to-unix-socket
- The value contains the path name to a UNIX domain socket where the flow
generator writes its data. The parent directory of
path-to-unix-socket must exist. Multiple probes may not use the
same path-to-unix-socket.
- poll-directory directory-path
- When this command is given, rwflowpack will periodically poll the
directory-path to look for files to process. flowcap will
exit with an error if you attempt to use probes that contain this command
since flowcap does not support reading data from files. Multiple
probes may not use the same directory-path. When polling the
directory, zero length files and files whose name begin with a dot
(".") are ignored. This command may be
used with the following probe types:
- For SiLK probes, each file must be a valid SiLK Flow file.
- IPFIX probes can process files created by the yaf program.
- A NetFlow v5 probe will process files containing NetFlow v5 PDUs. The
format of these files is specified in the description of the
read-from-file command.
- read-from-file dummy-value
- When this command is given, rwflowpack will read NetFlow v5 records
from the file specified by the --netflow-file command line switch.
The value to the read-from-file command is completely ignored, and
we recommend you use "/dev/null" as the
value. flowcap will exit with an error if you attempt to use probes
that contain this command since flowcap does not support reading
data from files. The format of a NetFlow v5 file is that the file's length
should be an integer multiple of 1464 bytes, where 1464 is the maximum
length of the NetFlow v5 PDU. Each 1464 block should contain the 24-byte
NetFlow v5 header and space for thirty 48-byte flow records, even if fewer
NetFlow records are valid. rwflowpack will accept NetFlow v5 files
that have been compressed with the gzip(1)
program.
- log-flags { none | { all | bad |
default | firewall-event | missing |
record-timestamps | sampling | show-templates | ... }
}
- This optional command accepts a comma- and/or space-separated list of
names that specify which messages to log for this probe. If not specified,
the default is default, which is equivalent to bad,
missing, sampling. The possible values are:
- all
- Log everything.
- bad
- Write messages about an individual NetFlow v5 record where the packet or
octet count is zero, the packet count is larger than the octet count, or
the duration of the flow is larger than 45 days.
- default
- Enable the following values: bad, missing, sampling.
This is the default value. Since SiLK 3.10.0. (Prior to SiLK
3.10.0, all was the default.)
- firewall-event
- When the "firewall-event" quirks flag is
set and the probe is processing NetFlow v9 or IPFIX, write messages about
records that are ignored because the firewall event information element on
the record is something other than flow deleted or flow denied. Since
SiLK 3.8.1.
- missing
- Examine the sequence numbers of NetFlow v5 packets and write messages
about missing and out-of-sequence packets. (You may suppress messages
regarding out-of-sequence NetFlow v9 or IPFIX packets for all
probes by setting the SILK_LIBFIXBUF_SUPPRESS_WARNINGS environment
variable.)
- none
- Log nothing. It is an error to combine this value with any other.
- record-timestamps
- Log the timestamps that appear on each record. This produces a lot of
output, and it is primarily used for debugging. Since SiLK
3.10.0.
- sampling
- Write messages constructed by parsing the NetFlow v9 Options Templates
that specify the sampling algorithm (when samplingAlgorithm and
samplingInterval IEs are present) or flow sampler mode (when
flowSamplerMode and flowSamplerRandomInterval IEs are present). Since
SiLK 3.8.0.
- show-templates
- Write messages to the log file describing each IPFIX template that is read
by this file-base or TCP probe. (UDP probes must still rely on the
SILK_IPFIX_PRINT_TEMPLATES environment variable.) The message contains
embedded new lines, with the template ID and domain on the first line, and
each of the template's elements on the following lines. Each element is
described by its name, its IE number with the private enterprise number if
any, and its length in the template. Scope elements in options templates
are marked. The format is that described in Section 10.2 of RFC7013
<https://tools.ietf.org/html/rfc7013>. Since SiLK
3.19.0.
- interface-values { snmp | vlan }
- This optional command specifies the values that should be stored in the
"input" and
"output" fields of the SiLK Flow records
that are read from the probe. If this command is not given, the default is
snmp. Note that NetFlow v5 probes only support snmp.
- snmp
- Store the index of the network interface card (ifIndex) where the
flows entered and left the router, respectively.
- vlan
- Store the VLAN identifier for the source and destination networks,
respectively. If only one VLAN ID is available,
"input" is set to that value and
"output" is set to 0.
This setting does not affect whether
rwflowpack(8) stores the
"input" and
"output" fields to its output files.
Storage of those fields is controlled by rwflowpack's
--pack-interfaces switch.
- quirks { none | { firewall-event | missing-ips
| nf9-out-is-reverse | nf9-sysuptime-seconds |
zero-packets ... } }
- This optional command is used to indicate that special (or quirky)
handling of the incoming data is desired. The value none disables
all quirks, and that is the default setting. If the value is not
none, it may be a list of one or more of the values specified below
separated by commas and/or spaces. Since SiLK 3.8.0.
- firewall-event
- Enable checking for firewall event information elements (IEs) when
processing IPFIX or NetFlow v9 flow records. This quirk must be enabled
when collecting data from a Cisco ASA. The IPFIX firewallEvent IE is 233.
The Cisco elements are NF_F_FW_EVENT (IE 40005) and NF_F_FW_EXT_EVENT (IE
33002). When this quirk is active, firewall events that match the value 2
(flow deleted) are categorized as normal flows, firewall events that match
the value 3 (flow denied) are usually put into one of non-routed types
(e.g., innull, outnull, see
packlogic-twoway(3) and
packlogic-generic (3) for details), and all other
firewall events values are dropped. (Note that a log message is generated
for these dropped records; to suppress these messages, use the
log-flags command.) When this quirk is not provided, SiLK handles
these records normally, which may result in duplicate flow records. (Prior
to SiLK 3.8, SiLK dropped all flow records that contained a firewall event
IE.) Since SiLK 3.8.0.
- missing-ips
- Store a flow record even when the record's NetFlow v9/IPFIX template does
not contain IP addresses. One change in SiLK 3.8.0 was to ignore flow
records that do not have a source and/or destination IP address; this
quirk allows one to undo the effect of that change. Since SiLK
3.8.1.
- nf9-out-is-reverse
- Change handling of the OUT_BYTES and OUT_PKTS information elements to
match that in libfixbuf prior to 1.8.0. Specifically, treat information
elements 23 and 24 (OUT_BYTES and OUT_PKTS in RFC3954) as
reverseOctetDeltaCount and reversePacketDeltaCount, respectively. Starting
with libfixbuf-1.8.0, those NetFlow v9 elements are mapped to
postOctetDeltaCount and postPacketDeltaCount, respectively. Since
SiLK 3.17.2.
- nf9-sysuptime-seconds
- Work around an issue with NetFlow v9 records created by some middleboxes
(e.g., SonicWall) where the sysUpTime field in the packet header is
reported in seconds instead of in milliseconds. The incorrect units cause
the time stamps on flow records to be future dated. In addition, adjust
the time fields on single packet flow records. Since SiLK
3.14.0.
- none
- Do not enable any quirks.
- zero-packets
- Enable support for flow records either that do not contain a valid packets
field, such as those from the Cisco ASA series of routers, or that have an
unusually large bytes-per-packet ratio. When this quirk is active, SiLK
sets the packet count to 1 when the incoming IPFIX or NetFlow v9 flow
record has a the packet count if 0. This quirk may modify the file format
used by rwflowpack for IPv4 records in order to support large
byte-per-packet ratios. Since SiLK 3.8.0.
- priority value
- This optional command is deprecated. It exists for backwards compatibility
and will be removed in the next major release.
To summarize the probe types and the input they can accept:
Probe Type Berkeley Directory UnixDomain Single
Socket Polling Socket File
========== ========== ========== ========== ==========
ipfix tcp/udp yes
netflow-v5 udp yes yes
netflow-v9 udp
sflow udp
silk yes
This subsection describes the syntax of a list of CIDR blocks, a list of IPset
file names, and a list of integers. These lists are used in the sensor block
and group block commands described below.
A group block (see "Group Block") allows you to assign
names to these lists. Once the name is defined, it may be referenced in
other lists of the same type by prepending the "at" character
("@") to the group's name.
The lists are:
- cidr-block-list
- A cidr-block-list (or ipblock-list) contains one or more CIDR blocks or
group references that represent an address space. Adjacent values in the
list may be separated by multiple whitespace (space or tab) characters
and/or a single comma. When IPv4 and IPv6 addresses combined, IPv4
addresses are mapped into the ::ffff:0:0/96 netblock. For lists containing
more than a few CIDR blocks, consider using an IPset list instead.
- ipset-list
- An ipset-list contains the path names of one or more binary IPset files or
group references. To create an IPset file, use the
rwsetbuild (1) tool. Each path name may be a
double-quoted string ("example"); the
quote characters are not necessary if the path name does not contain
whitespace or any special characters (single-quote
"'", double-quote
""", comma
",", or pound
"#"). Adjacent values in the list may be
separated by multiple whitespace (space or tab) characters and/or a single
comma. When multiple IPset files are specified, a new IPset is created in
memory and the contents of the files are merged into it.
rwflowpack(8) exits with an error if the IPset file
does not exist or does not contain any IP addresses. Since SiLK
3.10.0.
- interface-list
- An interface-list contains one or more integers between 0 and 65535,
inclusive, or group references or that represent SNMP interface indexes or
VLAN identifiers. Adjacent values in the list may be separated by multiple
whitespace (space or tab) characters and/or a single comma.
The use of group blocks is optional. They are a convenience to define and give a
name to a list of commonly used CIDR blocks, IPset files, or integer values
that are treated as SNMP interfaces or VLAN identifiers. Groups may be used in
sensor blocks ("Sensor Block") as described in the descriptions for
the discard-when, discard-unless,
network-name-ipblocks, network-name-ipsets and
network-name-interfaces commands, below.
The commands in a group definition must all be of the same type.
For example, you cannot mix ipblocks and ipsets commands in a
single group definition, even though both contain IP addresses.
The contents of an existing group may be added to the current
group block by using a group reference after the appropriate keyword as long
as both groups are the same type. A group reference is the name of
the group prefixed by the "at" character
("@"). When a group reference is used, the
contents of the existing groups are copied into the current group.
For examples of group blocks, see "Group definitions"
below.
The group command is used at top-level to begin a group
definition block, and the remaining commands are accepted within the group
block.
- group group-name
- The group command begins a new group definition block which
continues to the end group command. The argument to the
group command is the name of the group being defined. The
group-name must be unique among all groups. It must begin with a
letter, and it may not contain whitespace characters or the slash
character ("/").
- end group
- The end group command ends the definition of a group. Following an
end group command, top-level commands are again accepted.
- interfaces interface-list
- The interfaces command adds integer values to a group, where each
integer is treated as an SNMP interface number or VLAN identifier. An
interface-list is a list of integers or group references as defined
above ("Lists of CIDR Blocks, IPsets, or Integers"). The
interfaces command may appear multiple times in a group block.
- ipblocks cidr-block-list
- The ipblocks command adds CIDR block values to a group. The
cidr-block-list is described above. The ipblocks command may
appear multiple times in a group block.
- ipsets ipset-list
- The ipset command adds the IP addresses specified in an IPset file
to a group. The ipsets command may appear multiple times in a group
block. Since SiLK 3.10.0.
The information from the sensor block is used by rwflowpack to determine
how to categorize a flow; that is, in which file the flow record is stored.
The packlogic-twoway(3) manual page describes how
rwflowpack may use the sensor blocks to determine a record's category.
When the Sensor Configuration file is used with flowcap, no
sensors need to be defined. In fact, flowcap completely ignores all
text inside each sensor block.
The sensor block works with the packing logic to determine where
rwflowpack stores flow records. The packing logic plug-in specifies a
list of network names, and you will refer to these networks when you
configure the sensor block. Most plug-ins provide the
"external",
"internal", and
"null" names, where internal refers to
network being monitored, null are flows that were blocked by the router's
access control list, and external is everything else.
Several of the commands described below that categorize flow
records require as an argument a list of CIDR blocks, a list of IPset files,
or a list of integers. The syntax of these lists is described in the
"Lists of CIDR Blocks, IPsets, or Integers" section above.
As part of determining how to process a flow record,
rwflowpack may check a record's source or destination IP address
against a cidr-block-list or an ipset-set. Note the following:
- for a cidr-block-list, the IP address is sequentially compared to each
element of the list, stopping once a match is made
- when comparing an IPv4 address to an IPv6 list, the IPv4 address is
converted to IPv6 by mapping it into the ::ffff:0:0/96 prefix for purposes
of the comparison
- when comparing an IPv6 address to an IPv4 list, an IPv6 address in the
::ffff:0:0/96 prefix is converted to IPv4 for purposes of the comparison
and any other IPv6 address fails the comparison
As part of determining how to process a flow record,
rwflowpack may check whether the record's
"input" or
"output" fields are an interface-list.
Whether the "input" and
"output" fields contain SNMP interfaces or
VLAN identifiers is determined by the interface-values command in the
probe block (c.f. "Probe Block").
The sensor command is used in the top-level context to
begin a sensor configuration block, and the remaining commands are accepted
within the sensor block.
- sensor sensor-name
- The sensor command begins a new sensor configuration block. It
takes as an argument the name of the sensor being configured, and that
sensor must be defined in the Site Configuration file (see
silk.conf(5)). A sensor block is closed with the
end sensor command. You may have multiple sensor blocks that
have the same sensor-name.
- end sensor
- The end sensor command ends the configuration of a sensor.
Following an end sensor command, top-level commands are again
accepted.
- probe-type-probes probe-name [probe-name
...]
- This command associates the listed probe names of the given probe type
with the sensor. The probes do not have to be defined before they are
used. (Note this also means that a mistyped probe-name will not be
detected.) For example, netflow-v5-probes S1 says that S1 is a
netflow-v5 probe; whenever flow data arrives on the S1 probe, the sensor
associated with the probe notices that data is available and processes it.
Adjacent probe names in the argument list may be separated by space or tab
characters and/or a single comma.
- source-network network-name
- This command causes the sensor to assume that all flows originated from
the network named network-name. For example, if a sensor is
associated a probe that only monitors incoming traffic, you could use
"source-network external" to specify
that all traffic originated from the external network.
- destination-network network-name
- This command causes the sensor to assume that all flows were sent to the
network named network-name.
- network-name-ipblocks {cidr-block-list |
remainder}
- This command specifies the IP-space that is assigned to the network named
network-name. The value of the command can be the keyword
remainder or a cidr-block-list as defined above. When the value is
the keyword remainder, the IP-space for network-name is
conceptually all IPs not assigned to other networks on this sensor. The
remainder keyword may only appear one time within a sensor
block.
- network-name-ipsets {ipset-list |
remainder}
- This command specifies the IP-space that is assigned to the network named
network-name. The value of the command can be the keyword
remainder or an ipset-list as defined above. When the value is the
keyword remainder, the IP-space for network-name is
conceptually all IPs not assigned to other networks on this sensor. The
remainder keyword may only appear one time within a sensor
block.
- network-name-interfaces {interface-list |
remainder}
- This command specifies the SNMP interface index(es) or VLAN identifiers
that are assigned to the network named network-name. The value of
the command may be the keyword remainder or an interface-list as
defined above. When the value is the keyword remainder, the
interface list is computed by finding all interface values not assigned to
other networks on this sensor. The remainder keyword may only
appear one time within a sensor block.
- isp-ip ip-address [ip-address ...]
- This optional command may be used for a sensor that processes NetFlow
data. The value to the command is a list of IP addresses in dotted-decimal
notation, where the IPs are the addresses of the NICs on the router. For
traffic that doesn't leave the router (and thus was sent to the router's
null-interface), some packing-logic plug-ins use these IPs to distinguish
legitimate traffic for the router (e.g., routing protocol traffic, whose
destination address would be in this list) from traffic that violated the
router's access control list (ACL).
The following optional sensor block commands provide a way to
filter the flow records that rwflowpack packs for a sensor. Each
filter begins with either discard-when or discard-unless,
mentions a flow record field, and ends with a list of values.
The discard-when command causes the sensor to ignore the
flow record if the property matches any of the elements in the list. When a
match is found, rwflowpack immediately stops processing the record
for the current sensor and the flow is not packed for this sensor.
The discard-unless command causes the sensor to ignore the
flow record unless the property matches one of the elements in the
list. That is, the flow record is packed only if its property matches one of
the values specified in the list, and, when multiple discard-unless
commands are present, if the flow record matches the values specified in
each.
For each individual property, only one of discard-when or
discard-unless may be specified.
- discard-when source-interfaces
interface-list
- Instructs rwflowpack to discard a flow record for this sensor if
the value in the flow's "input" field is
listed in interface-list. When paired with VLAN tagging (see the
interface-values command in the probe block), this allows the
administrator to discard flows that have a specific VLAN tag. The commands
discard-when source-interfaces and discard-unless
source-interfaces may not be specified on the same sensor, but
other discard- commands are permitted.
- discard-unless source-interfaces
interface-list
- Instructs rwflowpack to discard the flow record for this sensor
unless the flow's "input" field is in
interface-list. That is, the flow record may be packed only if its
"input" field matches one of the values
specified in interface-list. When paired with VLAN tagging, this
allows one to discard flows that do not have a specific VLAN tag. The
commands discard-when source-interfaces and discard-unless
source-interfaces may not be specified on the same sensor, but other
discard- commands are permitted.
- discard-when destination-interfaces
interface-list
- Discards a flow for this sensor when the flow's
"output" field matches a value in
interface-list. May not appear in the same sensor block with
discard-unless destination-interfaces.
- discard-unless destination-interfaces
interface-list
- Discards a flow for this sensor unless the flow's
"output" field matches a value in
interface-list. May not appear in the same sensor block with
discard-when destination-interfaces.
- discard-when any-interfaces
interface-list
- Discards a flow for this sensor when either the flow's
"input" or its
"output" field matches a value in
interface-list. May not appear in the same sensor block with
discard-unless any-interfaces.
- discard-unless any-interfaces
interface-list
- Discards a flow for this sensor unless either the flow's
"input" or its
"output" field matches a value in
interface-list. May not appear in the same sensor block with
discard-unless any-interfaces.
- discard-when source-ipblocks
cidr-block-list
- Discards a flow for this sensor when the flow's source IP address,
"sIP", matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-unless source-ipblocks.
- discard-unless source-ipblocks
cidr-block-list
- Discards a flow for this sensor unless the flow's source IP address,
"sIP", matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-when source-ipblocks.
- discard-when destination-ipblocks
cidr-block-list
- Discards a flow for this sensor when the flow's destination IP address,
"dIP", matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-unless destination-ipblocks.
- discard-unless destination-ipblocks
cidr-block-list
- Discards a flow for this sensor unless the flow's destination IP address,
"dIP", matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-when destination-ipblocks.
- discard-when any-ipblocks cidr-block-list
- Discards a flow for this sensor when either the flow's source IP or its
destination IP address matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-unless any-ipblocks.
- discard-unless any-ipblocks
cidr-block-list
- Discards a flow for this sensor unless either the flow's source IP or its
destination IP address matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-when any-ipblocks.
- discard-when source-ipsets ipset-list
- Discards a flow for this sensor when the flow's source IP address,
"sIP", is in one of IPset files in
ipset-list. May not appear in the same sensor block with
discard-unless source-ipsets. Since SiLK 3.10.0.
- discard-unless source-ipsets ipset-list
- Discards a flow for this sensor unless the flow's source IP address,
"sIP", is in one of IPset files in
ipset-list. May not appear in the same sensor block with
discard-when source-ipsets. Since SiLK 3.10.0.
- discard-when destination-ipsets
ipset-list
- Discards a flow for this sensor when the flow's destination IP address,
"dIP", is in one of the IPset files in
ipset-list. May not appear in the same sensor block with
discard-unless destination-ipsets. Since SiLK
3.10.0.
- discard-unless destination-ipsets
ipset-list
- Discards a flow for this sensor unless the flow's destination IP address,
"dIP", is in one of the IPset files in
ipset-list. May not appear in the same sensor block with
discard-when destination-ipsets. Since SiLK
3.10.0.
- discard-when any-ipsets ipset-list
- Discards a flow for this sensor when either the flow's source IP or its
destination IP address is in one of the IPset files in ipset-list.
May not appear in the same sensor block with discard-unless
any-ipsets. Since SiLK 3.10.0.
- discard-unless any-ipsets ipset-list
- Discards a flow for this sensor unless either the flow's source IP or its
destination IP address is in one of the IPset files in ipset-list.
May not appear in the same sensor block with discard-when
any-ipsets. Since SiLK 3.10.0.
All these examples assume you are using the
packlogic-twoway(3) packing logic plug-in to
rwflowpack(8).
The following shows how to create groups that can be used in other group blocks
or in certain commands within a sensor block.
group G01
interfaces 1 2, 3
interfaces 4
end group
group G02
interfaces 5 @G01
end group
group G03
interfaces @G02
interfaces 6
end group
group G11
ipblocks 192.0.2.0/27 192.0.2.32/27, 192.0.2.64/26
ipblocks 192.0.2.128/25
end group
group G12
ipblocks 198.51.100.0/24 @G11
end group
group G13
ipblocks @G12
ipblocks 203.0.113.0/24
end group
group G21
ipsets /var/sets/ip1.set /var/sets/ip2.set, /var/sets/ip3.set
ipsets /var/sets/ip4.set
end group
group G22
ipsets /var/sets/ip5.set @G21
end group
group G23
ipsets @G22
ipsets /var/sets/ip6.set
end group
The following two blocks define a probe that listens on 9900/udp for NetFlow v5
from a router. The probe only accepts traffic originating from 172.16.22.22 or
172.16.33.33. The associated sensor uses the SNMP interfaces to categorize the
flows, where traffic that enters the router on interface 1 and leaves on
interface 8 is in, traffic entering on 8 and leaving on 1 is
out, traffic from 1 to 0 is innull, traffic from 8 to 8 is
int2int, etc.
probe S1 netflow-v5
listen-on-port 9901
protocol udp
accept-from-host 172.16.22.22 172.16.33.33
end probe
sensor S1
netflow-v5-probes S1
external-interfaces 1
internal-interfaces 8
null-interfaces 0
end sensor
The probe in this example is the same as above, except the administrator has
chosen to log only messages about bad packets (messages about missing packets
will be ignored). The sensor is categorizing flows by the source and
destination IP address in the flow record. The internal network is defined as
128.2.0.0/16, and all other IPs are defined as external. For example, HTTP
traffic whose source is 128.2.0.1 and destination is google.com will be
categorized as outweb; the reply (source of google.com and destination
128.2.0.1) will be inweb.
probe S2 netflow-v5
listen-on-port 9902
protocol udp
accept-from-host 172.16.22.22 172.16.33.33
log-flags bad # ignore missing pkts
end probe
sensor S2
netflow-v5-probes S2
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
This example uses an IPFIX probe to collect the flows on port 9903/tcp, where
the probe binds to address 192.168.1.92. The sensor configuration is the same
as in the previous example, but a group definition is used to define the
internal network.
probe S3 ipfix
listen-on-port 9903
protocol tcp
listen-as-host 192.168.1.92
end probe
group my-network
ipblocks 128.2.0.0/16
end group
sensor S3
ipfix-probes S3
internal-ipblocks @my-network
external-ipblocks remainder
end sensor
This example uses the same sensor configuration as above. The probe processes
files that have been created by yaf(1) and stored in the
directory /tmp/var/yaf/.
probe S4 ipfix
poll-directory /tmp/var/yaf
end probe
sensor S4
ipfix-probes S4
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
This example uses a NetFlow v9 probe to collect the flows on port 9905/udp,
where the probe binds to address 192.168.1.92. The sensor configuration uses
an IPset file to define the internal network.
probe S5 netflow-v9
listen-on-port 9905
protocol udp
listen-as-host 192.168.1.92
end probe
sensor S5
netflow-v9-probes S5
internal-ipsets /var/sets/my-network.set
external-ipsets remainder
end sensor
This example uses an sFlow probe to collect the flows on port 9906/udp, where
the probe binds to the IPv6 address ::1. The sensor configuration uses an
IPset file to define the internal network.
probe S19 sflow
listen-on-port 9906
protocol udp
listen-as-host ::1
end probe
sensor S19
sflow-probes S19
internal-ipsets /var/sets/my-network.set
external-ipsets remainder
end sensor
When collecting NetFlow v9 data from a Cisco ASA (Adaptive Security Appliance),
specify the quirks statement as shown in this example to enable special
handling of the NetFlow data.
probe S20 netflow-v9
listen-on-port 9988
protocol udp
quirks firewall-event zero-packets
end probe
sensor S20
netflow-v9-probes S20
internal-ipsets /var/sets/my-network.set
external-ipsets remainder
end sensor
Consider a scenario where there are multiple input streams that need to be
treated as a single sensor. For example, you use multiple routers for
load-balancing but you want them treated as a single logical sensor. In this
configuration, you send all the input streams to a single port, and you define
a single probe listening on that port. As long as the streams have a unique
source IP, the streams will be treated distinctly.
The following sensor and probe blocks accept any number of
TCP-based IPFIX connections to port 9907 and any number of NetFlow v5
connections to 9908. This configuration works for all types of input as SiLK
3.4.0 when using libfixbuf-1.2.0. See the configuration in the following
example for a alternate approach.
probe S7 ipfix
listen-on-port 9907
protocol tcp
end probe
sensor S7
ipfix-probes S7
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
probe S8 netflow-v5
listen-on-port 9908
protocol udp
log-flags bad
end probe
sensor S8
netflow-v5-probes S8
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
Like the previous example, this example configuration causes multiple input
streams to be treated as a single sensor. In this solution, each stream
arrives on a separate port where it is collected by a separate probe. The
sensor block combines the probes into one sensor. This type of approach works
with all types of input for all releases of SiLK.
probe S6-p1 netflow-v9
listen-on-port 9961
protocol udp
end probe
probe S6-p2 netflow-v9
listen-on-port 9962
protocol udp
end probe
probe S6-p3 netflow-v9
listen-on-port 9963
protocol udp
end probe
sensor S6
netflow-v9-probes S6-p1, S6-p2, S6-p3
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
Consider the case of using yaf on a monitor at the border of a network
where all traffic entering the network arrives at the monitor on one network
interface card (NIC) and all traffic leaving the network arrives at the
monitor on a different NIC. Since yaf does not support multiple
interfaces yet, you must run two yaf processes, one for each NIC. The
sensor configuration for this monitor would list two probes, each listening on
a different port, and two sensor blocks both packing to the same sensor. Each
sensor block packs the traffic as incoming or outgoing depending on which
probe received the traffic.
probe S9-in ipfix
listen-on-port 9991
protocol tcp
end probe
probe S9-out ipfix
listen-on-port 9992
protocol tcp
end probe
sensor S9
ipfix-probes S9-in
source-network external
destination-network internal
end sensor
sensor S9
ipfix-probes S9-out
source-network internal
destination-network external
end sensor
Suppose your network has multiple flow generators that you wish to treat as
separate sensors, but you would like to minimize the number of open ports on
your firewall. To support this configuration, configure the probes to
distinguish the traffic based on the source address. Specifically, create a
separate probe for each sensor where the probes of the same type use the same
listen-on-port value but different accept-from-host values.
(Different probe types may not bind the same port; the combination of
listen-on-port, protocol, and listen-as-host must be
unique for different probe types.) The following configuration uses a NetFlow
v5 probe, which works for all versions of SiLK. A similar configuration works
for any type of input as of SiLK 3.4.0 and libfixbuf-1.2.0.
probe S10 netflow-v5
listen-on-port 9910
accept-from-host 172.16.22.10
protocol udp
end probe
probe S11 netflow-v5
listen-on-port 9910
accept-from-host 172.16.22.11
protocol udp
end probe
group my-network2
ipblocks 128.2.0.0/16
end group
sensor S10
netflow-v5-probes S10
internal-ipblocks @my-network2
external-ipblocks remainder
end sensor
sensor S11
netflow-v5-probes S11
internal-ipblocks @my-network2
external-ipblocks remainder
end sensor
Suppose you have instrumented a single router but you wish to split the traffic
into two sensors, where one part of the network (monitored by sensor S12) is
defined as 128.2.0.0/17, and the other (sensor S13) as 128.2.128.0/17. Traffic
between 128.2.0.1 and google.com will be assigned to sensor S12, but it will
so appear as ext2ext traffic for sensor S13 unless you explicitly
discard that traffic using the discard-unless command.
probe S12-S13 ipfix
listen-on-port 9912
protocol tcp
end probe
group S12-space
ipblocks 128.2.0.0/17
end group
group S13-space
ipblocks 128.2.128.0/17
end group
sensor S12
ipfix-probes S12-S13
discard-unless any-ipblock @S12-space
internal-ipblocks @S12-space
external-ipblocks remainder
end sensor
sensor S13
ipfix-probes S12-S13
discard-unless any-ipblock @S13-space
internal-ipblocks @S13-space
external-ipblocks remainder
end sensor
You can configure rwflowpack to discard flows that do not have a
particular VLAN tag. First, specify the interface-values command to
instruct the probe to put the VLAN id into the fields that typically store the
SNMP interfaces. On the sensor, use the discard-unless command to
discard flows that do not have the desired VLAN tag (114 in this example).
Often you will not use the VLAN tags to determine a flow's direction
(category) since there is a single VLAN tag on each flow; instead, you specify
the IP space of the monitored network in the sensor block. (However, see the
next example.)
probe S14 ipfix
listen-on-port 9914
protocol tcp
interface-values vlan
end probe
sensor S14
ipfix-probes S14
discard-unless any-interface 114
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
By repeating a sensor block and using different discard-unless commands
in each block, you may configure rwflowpack to categorize flow records
based on VLAN tags. Suppose yaf is monitoring a connection where
incoming flows are marked with VLAN tag 151 and outgoing flows are marked with
152. You simply discard any traffic that does not have the wanted VLAN tag,
and use the source-network and destination-network commands to
assign the direction to the flow. In this example, any flow record that does
not have one of the expected VLAN tags has its source-network set to
"null", but since rwflowpack does not
expect a flow record to originate from the null network, it stores the record
in the other category for later analysis/debugging. (This example
requires SiLK 3.1 or later.)
probe S15 ipfix
listen-on-port 9915
protocol tcp
interface-values vlan
end probe
sensor S15
# vlan ID 151 is incoming
ipfix-probes S15
discard-unless source-interface 151
source-network external
destination-network internal
end sensor
sensor S15
# vlan ID 152 is outgoing
ipfix-probes S15
discard-unless source-interface 152
source-network internal
destination-network external
end sensor
sensor S15
# discard flows that have known IDs
# force unknown IDs into the "other" category
ipfix-probes S15
discard-when source-interface 151,152
source-network null
destination-network internal
end sensor
When yaf generates flow records from a multi-port Endace DAG card, it is
possible to use the port where the traffic was seen to categorize the traffic
in rwflowpack.
To do this, include the --dag-interface switch on the
yaf command line. This switch causes yaf to store the DAG port
where the packet was collected into the equivalent of the SNMP input field,
and yaf sets the SNMP output field to an offset of the port,
specifically the port plus 256 (0x100|port).
Assume DAG port 0 is connected to the external side of the network
(so it sees incoming traffic), and assume DAG port 1 is on the internal
side. For incoming traffic, yaf sets the input and output values to 0
and 256, respectively. For outgoing traffic, the values are 1 and 257.
The sensor.conf configuration file for rwflowpack
would be:
probe S16-dag ipfix
listen-on-port 9916
protocol tcp
end probe
sensor S16
ipfix-probes S16-dag
external-interface 0,257
internal-interface 1,256
end sensor
When rwflowpack processes the IPFIX flow records, it treats
flow records having an input of 0 and an output of 256 as traffic moving
from an external interface to an internal interface, and rwflowpack
packs those records as incoming. Similarly for the outgoing flow
records.
A probe whose type is "silk" must get its
flows by polling a directory of SiLK Flow files. The flows can be
re-categorized based on the IP addresses or based on the SNMP interfaces
(beware: often the SNMP interface values are 0 in SiLK Flow data). In this
example, the files in the directory /var/tmp/old-data/ are processed.
The internal network is defined as 128.2.0.0/16, and all other IPs are defined
as external.
probe S17 silk
poll-directory /var/tmp/old-data
end probe
sensor S17
silk-probes S17
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
Instead of listening on a UDP port for NetFlow traffic, you can configure the
probe to process a single file containing NetFlow v5 PDUs. This example
assumes you are running rwflowpack with the switches
--input-mode=file --netflow-file=FILENAME. The
--netflow-file switch overrides the read-from-file command on
the probe. rwflowpack will exit once it processes that single file.
probe S18 netflow-v5
log-flags bad # ignore missing pkts
read-from-file /dev/null # use --netflow-file=<file>
end probe
sensor S18
netflow-v5-probes S18
external-interface 182
internal-interface 189
null-interface 0
end sensor
rwflowpack(8), flowcap(8),
packlogic-twoway(3),
packlogic-generic(3), rwsetbuild(1),
silk.conf(5), silk(7), SiLK
Installation Handbook, pcap(3),
yaf(1), gzip(1)
Support for using double-quoted strings for IPset path names was added in SiLK
3.17.2.
The accept-from-host command began to accept a list of arguments
in SiLK 3.10.1.
SiLK 3.10.0 added IPset file support to the group block and
to some commands in the sensor block.
Support for collecting sFlow records was added in SiLK 3.9.0.
The quirks command was introduced in SiLK 3.8.0.
Visit the GSP FreeBSD Man Page Interface. Output converted with ManDoc. |