Manual Reference Pages - DIABLO-FILES (5)
diablo-files - Synopsis of configuration and control files for Diablo
NEWS SPOOL (/news/spool/news) (directory)
This manual page provides a synopsis for various diablo configuration
and control files. It does not necessarily describe all commands or
the specific file format. You can obtain most of the latter by looking
at the example config/control files in the
sample/ directory. What this manual page does is describe specific featurisms
that would otherwise not be readily apparent.
The location of all diablo files is controlled by /news/diablo.config.
The location of diablo.config itself can be specified with the
DIABLO_CONFIG_PATH environment variable or with the -C diabloconfigpath
option to any command. Diablo files are typically rooted at either
the news root, the library root, or the database root (path_news,
path_lib, or path_db in diablo.config). See samples/diablo.config
for more information.
dnewsfeeds file specifies how diablo generates outbound feeds as well as specifies
how diablo filters incoming feeds. You specify each outbound feed
label keyword, followed by various other keywords and ending with an
alias specifies an outbound newspath alias, which may contain * and ? wildcards.
An inbound article whos path element matches any given alias for any
given outbound feed is not requeued to that outbound feed.
delgroup filters articles to outbound feeds. All add/delgroup commands are run
against each newsgroup in the Newsgroups: header for the article and the
last add/delgroup command effecting any given newsgroup is applied to the
article. Normally this means you start out more general at the beginning
(i.e. *) and become more specific at the end of your list. When an
article is cross posted to multiple newsgroups, success with any of the
groups will cause the article to be propogated.
NOTE! If you addgroup *, then use delgroup and delgroupany to remove
groups that you do not want, beware that control messages for ALL groups
will still be propogated unless you also do a delgroup control.*.
delgroupany filters articles in a manner similar to delgroup, but if the specified groups
exist in the Newsgroups: line *AT ALL*, no distribution to *any* of the
other newsgroups will occur. i.e. if you say delgroupany alt.warez*, it
means that if an article is posted to comp.sys.amiga.misc and alt.warez*,
the article will NOT be propogated.
requiregroup An extremely exclusive filter that is the inverse of delgroupany. The
articles Newsgroups MUST contain a group matching the wildcard
or it will not be propogated. This is generally only used when splitting off
control feeds. Please see samples/dnewsfeeds for typical usage.
groupref allows a feed to recursively include a group access list defined by a groupdef
command. The groupdef command may occur before OR after the newsfeed that
references it, and groupdefs can be recursive. see
groupdef for more information.
filter effects INBOUND feeds. Articles coming FROM the specified label (see
diablo.hosts on how to tie labels to incomming connections) are rejected
if any listed newsgroup matches the wildcard specified in any of the
filter commands. You normally use this to reject articles cross posted
to your local newsgroups that are propogated from outside entities. Note
that the history file is not updated, just in case the article is also
brought in from a valid local newsfeed.
maxpath (1.08 or higher) effects OUTBOUND feeds. Diablo will not send an article
to an outbound feed if the articles Path: line contains more then the
specified number of path elements.
maxcross (1.04 or higher) effects OUTBOUND feeds. Diablo will not send an article
to an outbound feed if the articles Newsgroups: line contains more then
the specified number of groups.
maxsize (1.04 or higher) effects OUTBOUND feeds. Diablo will not send an article
to an outbound feed if the article is larger then the specified number of
bytes. You may use k, m, and g to denote kilobytes, megabytes,
and gigabytes. For example, <I>maxsize 100k</I> .
rtflush (1.12 or higher) effects OUTBOUND feeds. Diablo will write the queue file
for this feed unbuffered. Generally used along with realtime in
There is an (unimposed) arbitrary limit of 256 add/delgroup entries
per feed. Currently this is due to the fact that Diablo scans the
wildcard list linearly and cannot really support group-specific
wildcards (where a feed wants 10,000 specific groups rather then
fewer wildcarded groups).
A special label, called
ME may be specified. This label generally only contains
filter commands. Diablo will revert to this label for any incoming
connections that have not been associated with a specific label.
There is also the
groupdef command, which must occur outside any feed commands. The groupdef command
is followed by a set of groupadd, groupdel, groupdelany, and/or groupref
commands to create a grouplist. The groupdef command is terminated by an
end command. Grouplist definitions may be referenced by feeds via the
groupref command, and may be referenced by other grouplists.
dexpire.ctl file controls the length of time articles are stored for each group
in the reader header database and the maximum number of articles
that can be stored in the article index per group for dreaderd.
This index value is adjusted by dexpireover based on the number of
articles in the newsgroup.
The various options are separated by a colon and appear in any order
after a wildmat specification of the newsgroup names. The options are:
x to specify the number of days to keep articles matching this wildmat.
Not specifying this option means that articles are not removed. Fractions
of a day may also be used.
a to override the default 512 for the starting maxarts value per group
matching this wildmat. The starting value is only used when the group
index is first created.
i to specify the minimum maxarts that dexpireover may adjust to.
j to specify the maximum maxarts that dexpireover may adjust to.
dexpire.factor file is no longer used by Diablo and can be deleted. It may be used again
in the future.
dnntpspool.ctl file is used by the
dspoolout program to manage the outbound queues. Diablo will maintain a single
outbound queue file for each feed. dspoolout renames this file to a
proper sequenced queue file, properly flushes Diablos file descriptors,
starts up dnewslink programs, and removes any sequenced queue files
backlogged beyond the specified limit.
This file allows you to specify the remote host, maximum number of
queue files, and maximum number of dnewslink processes to run for
each outbound feed, and other options. Please refer to the
distrib.pats file is read and returned by the NNTP list distrib.pats command. It
is also used to generate a Distribution: header internally when the POST
command does not supply it (or supplies an empty Distributions header).
distributions file is read and returned by the NNTP list distributions command.
diablo.hosts file controls authentication for incoming connections. Each
line consists of a domain name or IP address wildcard and a label
identifying which feed in dnewsfeeds the incoming connection is
associated with. THE LABEL IS NO LONGER OPTIONAL. You must supply
Diablo normally requires that the reverse lookup match the forward
lookup for security purposes, but many sites set up their reverse
to point to a CNAME or set up their reverse to point to an unassociated
host yet still request that you put a common hostname in your hosts
file which resolves to all the IP addresses of their news machines.
Diablo will attempt to wildcard match the last two domain elements of
the reverse domain name with non-wildcard domain
names in diablo.hosts then issue a forward lookup of the name in
diablo.hosts and attempt to match the IP. In otherwords, if you have
an entry for newsfeeds.fubar.com in diablo.hosts and an incoming
connections reverse lookup comes back news55.fubar.com, diablo
will convert news55.fubar.com to *.fubar.com and attempt to match
that against entries in diablo.hosts. When a match occurs, it performs
a forward lookup (in this case against newsfeeds.fubar.com) and tries
to match up the IP address that way.
This methodology has the advantage of not requiring diablo to do a
sequential forward lookup of all the entries in diablo.hosts. Each
connections DNS load is consistent only with the domain/IP the connection
is coming from which is very important for stability in the face of a
large number of feeds.
dreaderd.hosts file contains access permissions for NNTP readers for the dreaderd server.
This file also contains access permissions for header-only feeds coming into
the dreaderd server.
dserver.hosts file contains the outgoing spool and posting server configuration which
dreaderd uses to make connections to outside servers for message retrieval
and outgoing POSTed message propogation.
dqueue directory contains the queue files, both the ones generated by the
diablo server and the ones maintained by dspoolout. Files in this
directory are generally named as the
label (diablo outbound queue file for a label), as the
label.Snnnnn (sequenced queue file maintained by dspoolout), or
.label.seq (sequence number information maintained by dspoolout).
Note that the diablo queue format has changed as of V1.08. Older versions
of diablo dumped the filename and message id. As of V1.08, an third field
formatted as OFFSET,BYTES is added to support the new multi-article spool
files. DNewslink understands both formats.
feeds directory contains automatic group add/del information as requested
by a feed through the
feedrset and other Diablo commands. If a file exists for any given feed,
it overrides the
delgroup commands in the dnewsfeeds file for that feed.
NEWS SPOOL (/news/spool/news) (directory)
Diablo implements a two-level news spool. A directory of the form D.xxxxxxxx
is created on the first level every 10 minutes. Each discrete fork creates
a distinct file in one or more of these directories when it receives an
article. The directory is chosen based on the expiration and the filename
is chosen starting with a hash of the incoming IP. The diablo process then
exclusively locks the file(s) in question. In the case of contention, the
loser will generate another filename and loop until it finds or creates one.
The files or of the form B.xxxx where xxxx is basically random. Multiple
articles may be stored in each file. An ascii NUL (code 0x00) is used as
an out-of-band article separator. The history and outgoing queue files
reference articles by relative file path, offset, and size.
It should be noted that dnewslink explicitly looks for the separator as a
double-check against corruption.
dactive.kp file is a key-token-pair database (see diablo-kp(5)). It is NOT compatible
with the INN active file. The dsyncgroups program with -G and -F flags may
be used to create dactive.kp from an active file in "list active" format
(see dsyncgroups(8)). A dactive.kp file may also be created based on an
active file on a remote host (also see dsyncgroups(8)).
Diablo KP database files are human readable but should only be manipulated
dkp program while Diablo is active. If Diablo is inactive, KP files may be
manipulated with dkp OR can be edited by hand. Diablos dactive.kp file
serves the same purpose as INNs active file but, being a general token=value
database, may contain additional information. The intention is to use this
database to track article number assigns and to store additional group
description, moderator email, and PGP keys for the reader portion of Diablo.
In Diablo, groups missing from dactive.kp do not normally effect the feeder
side of the system. However, Xref: headers are only generated for those
newsgroups listed in dactive.kp. This behavior can be changed through
the activedrop option in diablo.config
The dactive.kp database is keyed off the group name and uses the
following tokens, some of which are optional: NE (last stored article
number, %010d format), NB (first stored article number %010d format),
S (status), M (moderator email), and CPGP (pgp key, hierarchically
recursive). The status may contain multiple characters. n indicates
a disabled group, y indicates an enabled group, m indicates moderated.
itself is entirely optional (defaults to y). The group description, if any,
is stored with the GD key.
When using the feeder to generate Xref: headers, the feeder creates a copy
of the NE field called NX and uses that to track article number assignments.
This way both the reader and feeder can use the same physical active file
when both the reader and feeder are running on the same box. The dsyncgroups
command has options to manipulate NE and NX. WARNING! If NX is present, the
reader will reset NE if NX < NE and an incoming article has been assigned
a number less then NE.
if a group is marked moderated, the moderator email is obtained via the M key.
If control messages related to the group (hierarchically speaking), the CPGP
key contains the public key. For example, there might be an entry for comp
with a CPGP key but since comp is not a real newsgroup, the status would
Visit the GSP FreeBSD Man Page Interface.
Output converted with manServer 1.07.