Add the annotations requested by the
options to the transcript generated
--transcript option. Upper and lower case
options are treated identically. Available annotations are:
d Decoder diagnostics
p Parser warnings and error messages
w Most significant words and their probabilities
|As the dictionary is bring built by appending mail to it with the --mail and --junk options, unique words will automatically be pruned from it whenever the dictionary exceeds approximately n bytes. This is particularly handy when loading large collections of messages with --phrasemax set greater than one, as a very large number of unique phrases may clutter the dictionary being built and exceed the memory capacity of your computer. You could split the mail collection into multiple parts and explicitly --prune after each part, but --autoprune is much more convenient.|
|The frequency of words appearing in legitimate mail is inflated by the floating point factor n, which defaults to 2. This biases the classification of messages in favour of false negativesjunk mail deemed legitimate, while reducing the probability of false positives (legitimate mail erroneously classified as junk, which is bad). The higher the setting of --biasmail, the greater the bias in favour of false negatives will be.|
|Binary character streams (for example, attachments of application-specific files, including the executable code of worm and virus attachments) are scanned and contiguous sequences of alphanumeric ASCII characters n characters or longer are added to the list of words in the message. The dollar sign ($) is considered an alphanumeric character for these purposes, and words may have embedded hyphens and apostrophes, but may not begin or end with those characters. If --binword is set to zero, scanning of binary attachments is disabled entirely. The default setting is 5 characters.|
|The next --mail or --junk folder will be parsed using classic BSD rules for identifying the start of individual messages in the folder. In BSD-style folders, the text From as the leftmost characters of a line always denotes the start of a new message: any appearance of this text in any other context is always quoted, often by prefixing a > character. In the default Unix folder syntax, From only marks the start of a new message if it appears following one or more blank lines. Note that you must specify --bsdfolder before each folder to be read with BSD rules; it is not a modal setting.|
|Classify mail in fname. If it equals or exceeds the junk threshold (see --threshjunk), JUNK is written to standard output and the program exits with status code 3. If the message scores less than or equal to the mail threshold (see --threshmail), MAIL is written to standard output and the program exits with status 0. If the messages score falls between the two thresholds, its content is deemed indeterminate; INDT is written to standard output and the program exits with a status of 4. The output can be used to set an environment variable in Procmail to control the disposition of the message. If fname is - the message is read from standard input.|
|Clear appearances of words in junk mail from database. Used when preparing a database of legitimate mail.|
|Clear appearances of words in legitimate mail from database. Used when preparing a database of junk mail.|
|Print copyright information.|
|Import a dictionary from a comma-separated value (CSV) file fname. Records are assumed to be in the format written by --csvwrite but need not be sorted in any particular order. Words are added to those already in memory.|
|Export a dictionary as a comma-separated value (CSV) fname with this option. Such files can be loaded into spreadsheet or database programs for further processing. Words are sorted first in ascending order of probability they denote junk mail, then lexically.|
|--fread, -r fname|
|Load a fast dictionary (previously created with the --fwrite option) from file fname.|
|Write a dictionary to the file fname in fast dictionary format. Fast dictionaries are written in a binary format which is not portable across machines with different byte order conventions and cannot be added incrementally to assemble a larger dictionary, but can be loaded in a small fraction of the time required by the format created by the --write command. Using a fast dictionary for routine classification of incoming mail drastically reduces the time consumed in loading the dictionary for each message.|
|Print how-to-call information including a list of options.|
|--junk, -j fname|
|Add the mail in folder fname to the dictionary as junk mail. These folders may be compressed by a utility the host system can uncompress; specify the complete file name including the extension denoting its form of compression. If fname is - the mail folder is read from standard input.|
|--list||List the dictionary on standard output.|
|--mail, -m fname|
|Add the mail in folder fname to the dictionary as legitimate mail. These folders may be compressed by a utility the host system can uncompress; specify the complete file name including the extension denoting its form of compression. If fname is - the mail folder is read from standard input.|
|The probability that a word seen in mail which does not appear in the dictionary (or appeared too few times to assign it a probability with acceptable confidence) is indicative of junk is set to n. The default is 0.2the odds are that novel words are more likely to appear in legitimate mail than in junk.|
|Write a diagnostic file to the specified fname containing the actual lines the parser processed (after decoding of MIME parts and exclusion of data deemed unparseable). Use this option when you suspect problems in decoding or pre-parser filtering.|
|Limit the length of phrases assembled according to the --phrasemin and --phrasemax options to n characters. This permits ignoring phrases consisting of gibberish from mail headers and un-decoded content. In most cases these items will be discarded by a --prune in any case, but skipping them as they are generated keeps the dictionary from bloating in the first place. The default value is 48 characters.|
|Calculate probabilities of phrases consisting of a minumum of n words. The default of 1 calculates probabilities for single words.|
|Calculate probabilities of phrases consisting of a maximum of n words. The default of 1 calculates probabilities for single words. If you set this too large, the dictionary may grow to an absurd size.|
|After loading the dictionary, create a plot in fname .png of the histogram of words, binned by their probability of appearance in junk mail. In order to generate the histogram the GNUPLOT and NETPbm utilities must be installed on the system; if they are absent, the --plot option will not be available.|
|The POP3 proxy server activated by a subsequent --pop3server option will listen for connections on port n. If no --pop3port is specified, the server will listen on the default port of 9110. On most systems, youll have to run the program as root if you wish the proxy server to listen on a port numbered 1023 or less.|
|Activate a POP3 proxy server which relays requests made on the previously specified --pop3port or the default of 9110 if no port is specified, to the specified server, which may be given either as an IP address in dotted quad notion such as 10.89.11.131 or a fully-qualified domain name like pop.someisp.tld. The port on which the server listens for POP3 connections may be specified after the server prefixed by a colon (:) ; if no port is specified, the IANA assigned POP3 port 110 will be used. The POP3 proxy server will pass each message received on behalf of a requestor through the classifier and return the annotated transcript to the requestor, who may then filter it based on the classification appended to the message header. You must load a dictionary before activating the POP3 proxy server, and the --pop3server option must be the last on the command line. The server continues to run and service requests until manually terminated.|
|Write a trace of POP3 proxy server operations to standard error. Each trace message (apart from the dump of the body of multi-line replies to clients) is prefixed with the label POP3: .|
|--prune||After loading the dictionary from --mail and --junk folders, this option discards words which appear sufficiently infrequently that their probability cannot be reliably estimated. One usually --prune s the dictionary before using --write to save it for subsequent runs.|
|Include a token-by-token trace in the --pdiag output file. This helps when adjusting the parsers criteria for recognising tokens. Setting this option without also specifying a --pdiag file will have no effect other than perhaps to exercise your fingers typing it on the command line.|
|--read, -r fname|
|Load a dictionary (previously created with the --write option) from file fname.|
|The probability that a message is junk will be computed based on the individual probabilities of the n words with extremal probabilities; that is, probabilities most indicative of junk or mail. The default is 15, but theres no obvious optimal setting for this parameter; it depends in part on the average length of messages you receive.|
|To evade filtering programs, some junk mail is sent with MIME part headers which violate the standard but which most mail clients accept anyway. This option causes such messages to be parsed as a browser would, at the cost of standards compliance. If --sloppyheaders is used, it should be specified both when building the dictionary and when testing messages.|
|After loading the dictionary from --mail and --junk folders, print statistics of the distribution of junk probabilities of words in the dictionary. The statistics are written to standard output.|
|--test, -t fname|
|Test mail in fname and write the estimated probability it is junk to standard output unless the --transcript option is also specified with standard output (-) as the destination, in which case the inclusion of the probability and classification in the transcript is adjudged sufficient. If the --verbose option is specified, the individual probabilities of the most interesting words in the message will also be output. If fname is - the message is read from standard input.|
|Set the threshold for classifying a message as junk to the floating point probability value n. The default threshold is 0.9; messages scored above --threshjunk are deemed junk.|
|Set the threshold for classifying a message as legitimate mail to the floating point probability value n. The default threshold is 0.9, with messages scored below --threshmail deemed legitimate. Note that you may leave a gap between the --threshmail and --threshjunk values (although it makes no sense to set --threshmail higher). Mail scored between the two thresholds will then be judged of uncertain status.|
|Write an annotated transcript of the original message to the specified fname. If fname is -, the transcript is written to standard output. At the end of the message header, an X-Annoyance-Filter-Junk-Probability header item giving the computed probability and an X-Annoyance-Filter-Classification item which gives the classification of the message according to the --threshmail and --threshjunk settings; the classification is given as Mail, Junk, or Indeterminate.|
|Print diagnostic information as the program performs various operations.|
|Print program version information.|
Write a dictionary to the file
fname. The dictionary is written in a binary format which may be
loaded on subsequent runs with the
--read option. Binary
dictionary files are portable among machines with different
architectures and byte order.
The program exits with a status of 0 when processing is successfully completed, 1 when an error (I/O or file access in most cases) occurs, and 2 to indicate a command line syntax error. If the --classify option is specified, an exit status of 0 identifies the message tested as legitimate mail, 3 marks it as junk, and a status of 4 is returned for messages which cannot be confidently classified as either mail or junk.
Files are read or written as requested by options on the command line; all options which read or write files take a fname argument which gives the file name. The --classify, --junk, --mail, --test, and --transcript options interpret an argument of - as denoting standard input or output.
On systems which provide the required services and utilities, arguments to the --junk and --mail options may be compressed files or the name of a directory containing one or more messages which will be read as if logically concatenated. Messages in the directory may be compressed or uncompressed.
Error messages and diagnostic output generated when the --verbose option is specified are written to standard error.
Millions, doubtless. This is a program which must cope with whatever garbage is fed to it from mail folders, trying to make the best of it. When it messes up, your efforts in identifying the message which caused the problem and submitting a verbatim copy of it with your bug report are much appreciated.
Please report bugs to email@example.com and include annoyance-filter in the Subject line. Thanks in advance.
John Walker http://www.fourmilab.ch/
This software is in the public domain. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, without any conditions or restrictions. This software is provided as is without express or implied warranty.
gnuplot(1), gs(1), gzip(1), netpbm(1), procmail(1), xpdf(1)
annoyance-filter is written using the Literate Programming http://www.literateprogramming.com/ methodology; the user manual, program, and internal documentation are developed together, closely interlinked. Whenever the program is modified, the documentation is automatically updated, reducing the risk of divergence between what the manual says and what the program does.
This man page is intended as a reference for the command line options and most common applications of the program. For comprehensive documentation, including details of how to integrate annoyance-filter with the procmail mail processing system, please refer to the complete documentation published in PDF format, available on the Web at: http://www.fourmilab.ch/annoyance-filter/annoyance-filter.pdf
If you have downloaded the annoyance-filter source distribution, the corresponding version of annoyance-filter.pdf is included in the archive. You can read PDF files with Acrobat reader (a free download from http://www.adobe.com/acrobat/readstep.html) or the xpdf or Ghostscript (gs) utilities.
|-->||ANNOYANCE-FILTER (1)||4 AUG 2004|