|-i *directories and/or files* (input file)||This specifies the directories and/or files to index. Directories will be indexed recursively. This is typically specified in the configuration file with the IndexDir directive instead of on the command line. Use of this switch overrides the configuration file settings.|
|-S [fs|http|prog] (document source/access mode)||
This specifies the method to use for accessing documents to index.
Can be either fs for local indexing via the file system (the default),
http for spidering, or prog for reading documents from an external program.
Located in the conf directory are example configuration files that demonstrate indexing with the different document source methods.
See the SWISH-FAQ for a discussion on the different indexing methods, and the difference between spidering with the http method vs. using the file system method.
|-f *indexfile* (index file)||
If you are indexing, this specifies the file to save the generated index in,
and you can only specify one file. See also IndexFile in the configuration file.
If you are searching, this specifies the index files (one or more) to search from. The default index file is index.swish-e in the current directory.
|-c *file ...* (configuration files)||
Specify the configuration file(s) to use for indexing. This file contains many directives that
control how Swish-e proceeds.
See SWISH-CONFIG for a complete listing of configuration file directives.
If you specify a directory to index, an index file, or the verbose option on the command-line, these values will override any specified in the configuration file.
You can specify multiple configuration files. For example, you may have one configuration file that has common site-wide settings, and another for a specific index.
|-e (economy mode)||For large sites indexing may require more RAM than is available. The -e switch tells swish to use disk space to store data structures while indexing, saving memory. This option is recommended if swish uses so much RAM that the computer begins to swap excessively, and you cannot increase available memory. The trade-off is slightly longer indexing times, and a busy disk drive.|
|-l (symbolic links)||
Specifying this option tells swish to follow symbolic links when indexing.
The configuration file value FollowSymLinks will override the command-line value.
The default is not to follow symlinks. A small improvement in indexing time my result from enabling FollowSymLinks since swish does not need to stat every directory and file processed to determine if it is a symbolic link.
|-N path (index only newer files)||
The -N option takes a path to a file, and only files newer than the specified
file will be indexed. This is helpful for creating incremental indexes that is,
indexes that contain just files added since the last full index was created of all files.
Example (bad example)
This will index as normal, but only files with a modified date newer than index.swish-e will be indexed.
This is a bad example because it uses index.swish-e which one might assume was the date of last indexing. The problem is that files might have been added between the time indexing read the directory and when the index.swish-e file was created which can be quite a bit of time for very large indexing jobs.
The only solution is to prevent any new file additions while full indexing is running. If this is impossible then it will be slightly better to do this:
Then search with
or merge the indexes
**incremental index format only**
The -r option puts swish-e into removal mode. Any input files (given with -i
or the IndexDir parameter) are removed from an existing index.
would remove file.html from the existing index.
**incremental index format only**
The -u option puts swish-e into update mode. The timestamp of each input
file is compared against the corresponding file in the existing index.
If swish-e encounters an input file that either
does not exist yet in the index or exists with a timestamp older than the input file,
the input file is updated in the index. Any words in the input file
that have been added or removed are reflected as such in the index.
would update the index.swish-e index with the contents of file.html. If file.html was new, it would be added. If file.html already existed in the index, its contents would be updated in the index.
|-v [0|1|2|3] (verbosity level)||
The -v option can take a numerical value from 0 to 3.
Specify 0 for completely silent operation and 3 for detailed reports.
If no value is given then 1 is assumed. See also IndexReport in the configuration file.
Warnings and errors are reported regardless of the verbosity level. In addition, all error and warnings are written to standard out. This is for historical reasons (many scripts exist that parse standard out for error messages).
|-W (0|1|2|3) (parser warning level)||
If using the libxml2 parser, the default parser warning level is set at 2. Use the -W
option to override that default. Most often, you might want to turn it off altogether:
would fail silently if the parser encountered any errors.
The following command line arguments are available when searching with Swish-e. These switches are used to select the index to search, what fields to search, and how and what to print as results.
This section just lists the available command line arguments and their usage. Please see SWISH-SEARCH for detailed searching instructions.
Warning: If using Swish-e via a CGI interface, please see CGI Danger!
Security Note: If the swish binary is named swish-search then swish will not allow any operation that would cause swish to write to the index file.
-w *word1 word2 ...* (query words) This performs a case-insensitive search using a number of keywords. If no index file to search is specified (via the -f switch), swish-e will try to search a file called index.swish-e in the current directory.
swish-e -w word
Phrase searching is accomplished by placing the quote delimiter (a double-quote by default) around the search phrase.
swish-e -w word or "this phrase"
Search would should be protected from the shell by quotes. Typically, this is single quotes when running under Unix.
Under Windows command.com you may not need to use quotes, but you will need to backslash the quotes used to delimit phrases:
swish-e -w \"a phrase\"
The phrase delimiter can be set with the -P switch.
The search may be limited to a MetaName. For example:
swish-e -w meta1=(foo or baz)
will only search within the meta1 tag.
Please see SWISH-SEARCH for a description of MetaNames
-f *file1 file2 ...* (index files) Specifies the index file(s) used while searching. More than one file may be listed, and each file will be searched. If no -f switch is specified then the file index.swish-e in the current directory will be used as the index file. -m *number* (max results) While searching, this specifies the maximum number of results to return. The default is to return all results.
This switch is often used in conjunction with the -b switch to return results one page at a time (strongly recommended for large indexes).
-b *number* (beginning result) Sets the begining search result to return (records are numbered from 1). This switch can be used with the -m switch to return results in groups or pages.
swish-e -w word -b 1 -m 20 # first page swish-e -w word -b 21 -m 20 # second page
-t HBthec (context searching) The -t option allows you to search for words that exist only in specific HTML tags. Each character in the string you specify in the argument to this option represents a different tag in which to search for the word. H means all HEAD tags, B stands for BODY tags, t is all TITLE tags, h is H1 to H6 (header) tags, e is emphasized tags (this may be B, I, EM, or STRONG), and c is HTML comment tags
search only in header (<H*>) tags
swish-e -w word -t h
-d *string* (delimiter) Set the delimiter used when printing results. By default, Swish-e separates the output fields by a space, and places double-quotes around the document title. This output may be hard to parse, so it is recommended to use -d to specify a character or string used as a separator between fields.
The string dq means double-quotes.
swish-e -w word -d , # single char swish-e -w word -d :: # string swish-e -w word -d " # double quotes under Unix swish-e -w word -d \" # double quotes under Windows swish-e -w word -d dq # double quotes
The following control characters may also be specified: \t \r \n \f.
Warning: This string is passed directly to sprintf() and therefore exposes a securty hole. Do not allow user data to set -d format strings directly.
-P *character* Sets the delimiter used for phrase searches. The default is double quotes ".
Some examples under bash: (be careful about you shell metacharacters)
swish-e -P ^ -w title=^words in a phrase^ swish-e -P \ -w "title=words in a pharse"
-p *property1 property2 ...* (display properties) This causes swish to print the listed property in the search results. The properties are returned in the order they are listed in the -p argument.
Properties are defined by the ProperNames directive in the configuration file (see SWISH-CONFIG) and properties must also be defined in MetaNames. Swish stores the text of the meta name as a property, and then will return this text while searching if this option is used.
Properties are very useful for returning data included in a source documnet without having to re-read the source document while searching. For example, this could be used to return a short document description. See also see Document Summeries and PropertyNames in SWISH-CONFIG.
To return the subject and category properties while indexing.
swish-e -w word -p subject category
Properties are returned in double quotes. If a property contains a double quote it is HTML escaped ("). See the -x switch for a more advanced method of returning a list of properties.
NOTE: it is necessary to have indexed with the proper PropertyNames directive in the user config file in order to use this option.
-s *property [asc|desc] ...* (sort) Normally, search results are printed out in order of relevancy, with the most relevant listed first. The -s sort switch allows you to sort results in order of a specified property, where a property was defined using the MetaNames and PropertyNames directives during indexing (see SWISH-CONFIG).
The string passed can include the strings asc and desc to specify the sort order, and more than one property may be specified to sort on more than one key.
sort by title property ascending order
sort descending by title, ascending by name
-s title desc name asc
Note: Swish limits sort keys to 100 characters. This limit can be changed by changing MAX_SORT_STRING_LEN in src/config.h and rebuilding swish-e.
-L limit to a range of property values (Limit) This is an experimental feature!
The -L switch can be used to limit search results to a range of property values
swish-e -w foo -L swishtitle a m
finds all documents that contain the word foo, and where the documents title is in the range of a to m, inclusive. By default, the case of the property is ignored, but this can be changed by using PropertyNamesCompareCase configuation directive.
Limiting may be done with user-defined properties, as well.
For example, if you indexed documents that contain a created timestamp in a meta tag:
<meta name="created_on" content="982648324">
Then you tell Swish that you have a property called created_on, and that its a timestamp.
After indexing you will be able to limit documents to a range of timestamps:
-w foo -L created_on 946684800 949363199
will find documents containing the word foo and that have a created_on date from the start of Jan 1, 2000 to the end of Jan 31, 2000.
Note: swish currently does not parse dates; Unix timestamps must be used.
Two special formats can be used:
-L swishtitle <= m -L swishtitle >= m
Finds titles less than or equal, or grater than or equal to the letter m.
This feature will not work with swishrank or swishdbfile properties.
This feature takes advantages of the pre-sorted tables built by swish during indexing to make this feature fast while searching. You should see in the indexing output a line such as:
6 properties sorted.
That indicates that six pre-sorted tables were built during indexing. By default, all properties are presorted while indexing. What properties are pre-sorted can be controlled by the configuration parameter PreSortedIndex.
Using the -L switch on a property that was not pre-sorted will still work, but may be much slower during searching.
Note that the PropertyNamesSortKeyLength setting is used for sorting properties. Using too small a PropertyNamesSortKeyLength could result in -L selecting the wrong properties due to incomplete sorting.
This is an experimental feature, and its use and interface are subject to change.
-x formatstring (extended output format) The -x switch defines the output format string. The format string can contain plain text and property names (including swish-defined internal property names) and is used to generate the output for every result. In addition, the output format of the property name can be controlled with C-like printf format strings. This feature overrides the cmdline switches -d and -p, and a warning will be generated if -d or -p are used with -x.
Warning: The format string (fmt) is passed directly to sprintf() and therefore exposes a securty hole. Do not allow user data to set -x format strings directly.
For example, to return just the title, one per line, in the search results:
swish-e -w ... -x <swishtitle>\n ...
Note: the \n may need to be protected from your shell.
See also ResultExtFormatName for a way to define named format strings in the swish configuration file.
Format of formatstring:
Where propertyname is:
* the name of a user property as specified with the config file directive PropertyNames * the name of a swish Auto property (see below). These properties are defined automatically by swish you do not need to specify them with PropertyNames directive. (This may change in the future.)
propertynames must be placed within "< and >".
Swish-e allows you to specify certain META tags within your documents that can be used as document properties. The contents of any META tag that has been identified as a document property can be returned as part of the search results. Doucment properties must be defined while indexing using the PropertyNames configuration directive (see SWISH-CONFIG).
Examples of user-defined PropertyNames:
<keywords> <author> <deliveredby> <reference> <id>
Swish defines a number of Auto properties for each document indexed. These are available for output when using the -x format.
Name Type Contents -------------- ------- ---------------------------------------------- swishreccount Integer Result record counter swishtitle String Document title swishrank Integer Result rank for this hit swishdocpath String URL or filepath to document swishdocsize Integer Document size in bytes swishlastmodified Date Last modified date of document swishdescription String Description of document (see:StoreDescription) swishdbfile String Path of swish database indexfile
The Auto properties can also be specified using shortcuts:
Shortcut Property Name -------- -------------- %c swishreccount %d swishdescription %D swishlastmodified %I swishdbfile %p swishdocpath %r swishrank %l swishdocsize %t swishtitle
For example, these are equivalent:
-x <swishrank>:<swishdocpath>:<swishtitle>\n -x %r:%p:%t\n
Use a double percent sign %% to enter a literal percent sign in the output.
Formatstrings of properties:
Properties listed in an -x format string can include format control strings. These propertyformats are used to control how the contents of the associated property are printed. Property formats are used like C-language printf formats. The property format is specified by including the attribute fmt within the property tag.
Format strings cannot be used with the % shortcuts described above.
-x <propertyname fmt="propfmtstr">
where subfmt controls the output format of propertyname.
Examples of property format strings:
date type: <swishlastmodified fmt="%d.%m.%Y"> string type: <swishtitle fmt="%-40.35s"> integer type: <swishreccount fmt=/%8.8d/>
Please see the manual pages for strftime(3) and sprintf(3) for an explanation of format strings. Note: some versions of strftime do not offer the %s format string (number of seconds since the Epoch), so swish provides a special format string %ld to display the number of seconds since the Epoch.
The first character of a property format string defines the delimiter for the format string. For example,
-x "<author fmt=[%20s]> ...\n" -x "<author fmt=%20s> ...\n" -x "<author fmt=/%20s/> ...\n"
Standard predefined formats:
If you ommit the sub-format, the following formats are used:
String type: "%s" (like printf char *) Integer type: "%d" (like printf int) Float type: "%f" (like printf double) Date type: "%Y-%m-%d %H:%M:%S" (like strftime)
Text in formatstring or propfmtstr:
Text will be output as-is in format strings (and property format strings). Special characters can be escaped with a backslash. To get a new line for each result hit, you have to include the Newline-Character \n at the end of fmtstr.
-x "<swishreccount>|<swishrank>|<swishdocpath>\n" -x "Count=<swishreccount>, Rank=<swishrank>\n" -x "Title=\<b\><swishtitle>\</b\>" -x Date: <swishlastmodified fmt="%m/%d/%Y">\n -x Date in seconds: <swishlastmodified fmt=/%ld/>\n
you can use C-like control escapes in the format string:
known controls: \a, \b, \f, \n, \r, \t, \v, digit escapes: \xhexdigits \0octaldigits character escapes: \anychar
swish -x "%c\t%r\t%p\t\"<swishtitle fmt=/%40s/>\"\n"
Examples of -x format strings:
-x "%c|%r|%p|%t|%D|%d\n" -x "%c|%r|%p|%t|<swishdate fmt=/%A, %d. %B %Y/>|%d\n" -x "<swishrank>\t<swishdocpath>\t<swishtitle>\t<keywords>\n -x "xml_out: \<title\><swishtitle>\>\</title\>\n" -x "xml_out: <swishtitle fmt=<title>%s</title>>\n"
-H [0|1|2|3|<n>] (header output verbosity) The -H n switch generates extened header output. This is most useful when searching more than one index file at a time by specifying more than one index file with the -f switch. -H 2 will generate a set of headers specific to each index file. This gives access to the settings used to generate each index file.
Even when searching a single index file, -H n will provided additional information about the index file, how it was indexed, and how swish is interperting the query.
-H 0 : print no header information, output only search result entries. -H 1 : print standard result header (default). -H 2 : print additional header information for each searched index file. -H 3 : enhanced header output (e.g. print stopwords). -H 9 : print diagnostic information in the header of the results (changed from: C<-v 4>)
-R [0|1] (Ranking Scheme) This is an experimental feature!
The default ranking scheme in SWISH-E evaluates each word in a query in terms of its frequency and position in each document. The default scheme is 0.
New in version 2.4.3 you may optionally select an experimental ranking scheme that, in addition to document frequency and position, uses Inverse Document Frequency (IDF), or the relative frequency of each word across all the indexes being searched, and Relative Density, or the normalization of the frequency of a word in relationship to the number of words in the document.
NOTE: IgnoreTotalWordCountWhenRanking must be set to no or 0 in your index(es) for -R 1 to work.
Specify -R 1 to turn on IDF ranking. See the API documentation for how to set the ranking scheme in your Perl or C program.
-V (version) Print the current version. -k *letter* (print out keywords) The -k switch is used for testing and will cause swish to print out all keywords in the index beginning with that letter. You may enter -k * to generate a list of all words indexed by swish. -D *index file* (debug index) The -D option is no longer supported in version 2.2. -T *options* (trace/debug swish) The -T option is used to print out information that may be helpful when debugging swish-es operation. This option replaced the -D option of previous versions.
Running -T help will print out a list of available *options*
In previous versions of Swish-e indexing would require a very large amount of memory and the indexing process could be very slow. Merging provided a way to index in chunks and then combine the indexes together into a single index.
Indexing is much faster now and uses much less memory, and with the -e switch very little memory is needed to index a large site.
Still, at times it can be useful to merge different index files into one file for searching. This could be because you want to keep separate site indexes and a common one for a global search, or you have separate collections of documents that you wish to search all at one time, but manage separately.
-M *index1 index2 ... indexN out_index Merges the indexes specified on the command line the last file name entered is the output file. The output index must not exist (otherwise merge will not proceed).
Only indexes that were indexed with common settings may be merged. (e.g. dont mix stemming and non-stemming indexes, or indexes with different WordCharacter settings, etc.).
Use the -e switch while merging to reduce memory usage.
Merge generates progress messages regardless of the setting of -v.
-c *configuration file* Specify a configuration file while indexing to add administrative information to the output index file.
$Id: SWISH-RUN.pod 1741 2005-05-17 02:22:40Z karman $