GSP
Quick Navigator

Search Site

Unix VPS
A - Starter
B - Basic
C - Preferred
D - Commercial
MPS - Dedicated
Previous VPSs
* Sign Up! *

Support
Contact Us
Online Help
Handbooks
Domain Status
Man Pages

FAQ
Virtual Servers
Pricing
Billing
Technical

Network
Facilities
Connectivity
Topology Map

Miscellaneous
Server Agreement
Year 2038
Credits
 

USA Flag

 

 

Man Pages


Manual Reference Pages  -  SAVELOGS (1)

.ds Aq ’

NAME

savelogs - save/rotate/delete log files nicely

CONTENTS

SYNOPSIS

<B>savelogsB> saves your log files in a nice way (by default).



    savelogs --postmovehook=/usr/local/bin/restart_apache \
             --apacheconf=/www/conf/httpd.conf /var/log/messages

    savelogs `cat list_of_logs_to_process.txt`

    savelogs --loglevel=2 /var/log/maillog /var/log/cronlog \
             /var/log/messages

    savelogs --config=/etc/savelogs.conf

    savelogs --period=15 /var/log/messages

    savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com



DESCRIPTION

<B>savelogsB> is a flexible and robust log file archival system. Its logic is simple: move (rename) the log file, filter data from the log file, store the log file in an archive (via tar or gtar), and compress the archive (via gzip or compress). After successful compression, the original log file is deleted.

All of the above phases are optional. This means that you may simply delete files if you wish. Or you may simply compress existing log files. Or you may move files and add them to a tar file but leave the tar file uncompressed, etc. You pick ;o)

(If you just want to cut to the chase and don’t care how <B>savelogsB> works, see the EXAMPLES section near the bottom of this document.)

    Savelogs Phases

The processing order may be abbreviated into these five phases:



    move -> filter -> archive -> compress -> delete



any of which may be subtracted from the process order. In addition to these phases are some intermediate ’hooks’ where you may supply an external program or shell command to invoke. This is useful if you’re rotating web server log files and you need to HUP (restart) your web server after moving the logs (for example).

Subtracting phases is done in one of two possible ways. The first way is to specify it in the configuration file:



    Process          move,archive,delete



which will move log files and archive them (but not filter or compress them). After successful archival, the original log files will be deleted.

The second way is to specify it on the command-line:



    --process=compress,delete



which will simply compress log files (but not move, filter, or archive them).

In addition to the five phase processing options above, you may also employ the following abbreviations:
(no option specified) If you specify no <B>processB> option, the default is move,compress.
none Do none of the phases. This isn’t a very useful option.
all Do all of the phases.

    An Overview

A typical <B>savelogsB> session might begin by typing this command:



    savelogs /var/log/messages



After which the following phases will execute:
move The log file is renamed:



    /var/log/messages --> /var/log/messages.010523



compress The log file is compressed



    /var/log/messages.010523 --> /var/log/messages.010523.gz



    A Word About Paths

All paths you specify to <B>savelogsB> should be relative to your home directory (if you’re the root user—uid 0--your home directory is set to ’/’). You do not need to use a tilde (~). You may assume that <B>savelogsB> runs from your home directory and knows how to handle absolute paths.

If my real home directory were located in /usr/home/joe and I wanted to rotate the log file located in /usr/home/joe/var/log/messages, I would do something like this:



    savelogs /var/log/messages



and <B>savelogsB> would Do What I Mean.

The only exception to this are external commands given to <B>postmovehookB>, <B>postfilterhookB> and other such options. Paths you specify here really are full paths.

CONFIGURATION

    Configuration file option format

<B>savelogsB> will read its configuration options from a configuration file (which you must supply) or from the command-line. Creating a configuration file is easy: it is a simple Apache-style plaintext file that has options specified in this format:



    Option          value



where Option is one of the options below and value is either a true/false; yes/no; on/off combination or some string value, such as a pathname and file (depending on the nature of the option).

Your distribution of <B>savelogsB> may have included a sample configuration file for you to edit as you wish in ~/etc/savelogs.conf.sample.

    Option processing order

Configuration options are first read from <B>savelogsB> internal defaults, which are sprinkled throughout this document. Next <B>savelogsB> reads its configuration file, if any is specified. Lastly, <B>savelogsB> uses options from the command-line.

For example, the default value for the <B>periodB> directive is 10. If you ran <B>savelogsB> like this:



    savelogs --period /var/log/messages



every day for 10 days, you would have 10 archived log files.

If you have in your configuration file:



    Period            5



and ran the same command above every day for 10 days, you’d only have 5 archived log files because the configuration file overrides <B>savelogsB> internal defaults. Finally, if you had a configuration file with the previously mentioned value for <B>periodB> and ran this command:



    savelogs --period=7 /var/log/messages



every day for 10 days, you would have 7 archived log files because command-line options override configuration file options (which override internal default values).

    Available options

All options you may specify on the command-line you may also sepcify in a configuration file (except the configuration file directive itself). For example, if you had a cronjob that did this:



    savelogs --process=delete \
    --postmovehook="/usr/local/apache/bin/apachectl graceful" \
    --apacheconf=/www/conf/httpd.conf



(which deletes all apache logs) you could make a nice configuration file (call it ~/etc/savelogs1.conf) that would do the same thing:



    Process         delete
    PostMoveHook    /usr/local/apache/bin/apachectl graceful
    ApacheConf      /www/conf/httpd.conf



and then invoke your cron like this:



    savelogs --config=/etc/savelogs1.conf



    Case-sensitivity

A sample configuration file may be located in ~/etc/savelogs.conf.sample which you may copy and edit as you wish. Configuration file directives are case-insensitive:



    Process    move,compress
    PROCESS    move,compress
    process    move,compress
    pROceSs    move,compress



are all the same directive to savelogs. Configuration file values ARE case-sensitive.



    ApacheConf    /www/conf/httpd.conf
    ApacheConf    /WWW/conf/httpd.conf
    ApacheConf    /www/CONF/httpd.conf
    ApacheConf    /www/conf/Httpd.conf



are four distinct files, depending on the case-sensitivity of your operating system.

    Testing configurations

When you are testing new configuration directives, use the <B>dry-runB> option and watch the log output using the <B>loglevelB> and <B>logfileB> directives. This will help you avoid losing data unnecessarily.

You may also specify the <B>settingsB> option which will show you all the current settings after defaults, configuration file, and command-line options have been processed.

OPTIONS

Options given below are configuration directives that may appear in a configuration file or on the command-line. Options are case-insensitive, i.e., ApacheLog is the same as apachelog, though the values associated with the options are often case-sensitive (e.g., paths, filenames, etc.)

Options specified on the command-line should be prefixed with two hyphens. Some options do not make sense in a configuration file or need to occur before the configuration file is parsed such as <B>configB>, <B>helpB>, or <B>homeB>.

    Running Savelogs

<B>helpB> Shows a brief usage statement and exits.

Example:



    savelogs --help



<B>versionB> Shows the current version of <B>savelogsB> and exits.

Example:



    savelogs --version



<B>settingsB> Shows the current settings of <B>savelogsB> and exits.

Example:



    savelogs --settings --apacheconf=/www/conf/httpd.conf \
             /var/log/messages



<B>dry-runB> When used with the <B>logfileB> and <B>loglevelB> settings, <B>dry-runB> will show you what will happen if you were to run <B>savelogsB> with the current settings without actually doing it. This is a useful option to specify if you want to see what effect some changes might have, or to see which files are going to get archived with the current settings.

Note that <B>savelogsB> running under the <B>dry-runB> directive will sometimes produce errors that wouldn’t occur during normal use. This can happen for a variety of reasons, mostly related to <B>savelogsB> looking for files that don’t yet exist, or archives that don’t yet exist because they weren’t actually created. In this respect, <B>dry-runB> doesn’t give you precisely what will happen, but it does give you a good idea. Use it with a grain of salt.

Example:



    savelogs --dry-run --loglevel=2 /var/log/foo



<B>homeB> Changes the default base location of where <B>savelogsB> runs from. This is mostly a debugging utility. Consider this an advanced feature which should probably be ignored. Defaults to the process owner’s home directory (which is almost always what you want).

Example:



    Home                /usr/home/joe/usr/home/bob



<B>configB> Changes the default configuration file <B>savelogsB> reads at startup. This should be done from the command-line or it won’t have any effect.

Example:



    savelogs --config=/etc/my_other_savelogs.conf



<B>process=[move],[filter],[archive],[compress],[delete]B> Tells <B>savelogsB> which phases to execute. If you just want to move (rename) logs, do this:



    savelogs --process=move /var/log/messages



and ~/var/log/messages will become ~/var/log/messages.yymmdd (where yymmdd are today’s date).

For just removing logs, specify the delete option. You can get fancy:



    savelogs --process=move,compress /var/log/messages



which renames the log file ~/var/log/messages to ~/var/log/messages.yymmdd and then compresses it, not filtering, archiving (i.e. putting into a separate tar file), or deleting it (the compress option also renames the file so that delete becomes useless since the file as it was no longer exists).

move,compress is the default value for the <B>processB> option, so the above directive could have also been specified:



    savelogs /var/log/messages



You may also specify <B>allB> or <B>noneB> as shortcuts for



    savelogs --process=move,filter,archive,compress,delete



and



    savelogs --process=



respectively.

    Savelogs Logging

<B>savelogsB> has an internal logging facility that helps you diagnose problems, or just see what’s going on. By default, <B>savelogsB> writes to stdout (i.e., your screen if you’re running this from a tty).
<B>loglevel=#B> Determines how verbose <B>savelogsB> is when it’s writing its own internal messages. Valid values are between 0 and 5 inclusive.

Example:



    LogLevel            3



The general rule of thumb for <B>savelogsB> logging is this:



    Level  What will be logged
    =====  ===================
    0      no output except fatal errors
    1      Level 0 + start/finish stats, errors
    2      Level 1 + warnings, logfiles to process
    3      Level 2 + chdir, filter, phase completion
    4      Level 3 + phase core actions, phase beginning
    5      Level 4 + everything else



The first few times you run <B>savelogsB>, try a higher <B>loglevelB> value to see what’s happening with your log files. Once you’re comfortable with how <B>savelogsB> works, you may turn it down a few notches (level 1 is usually fine) so at least you can check to see if your cronjob actually ran ;o)

<B>logfile=[stdout|stderr|/path/to/log]B> <B>savelogsB>, depending on the <B>loglevelB> you’ve specified, writes what it’s doing, such as moving, archiving, or deleting, etc. The <B>logfileB> directive tells <B>savelogsB> where to write all these messages.

The default value for <B>logfileB> is stdout which means that your output will go to the screen unless you’ve redirected stdout. You may also specify stderr or the path to a file where you’d like messages to be appended.

Example:



    LogFile             /var/log/savelogs.log



    How Savelogs Finds Logs to Process

<B>savelogsB> processes the logs you specify on the command-line (items on the command-line that are not recognized as options are assumed to be log files to process).

If no logs are specified (either on the command-line or in a configuration file using the following directives), <B>savelogsB> will complain and show a ’usage’ statement. To turn off the usage statement, use the <B>gripeB> directive (to gripe is the default behavior):



    savelogs --nogripe /no/such/log



To save wear on your finger tips and phosphor in your monitor, we recommend liberal use of the following configuration directives.
<B>Log=/path/to/logB> This works just like adding a file on the command-line, but is included so that you can put log files you want processed in a configuration file. It will also work on the command-line:

Example:



    savelogs --log=/var/log/messages



is the same as:



    savelogs /var/log/messages



which is also equivalent to a config file named ’~/etc/savelogs.conf’ with this single line:



    Log                  /var/log/messages



and invoked like this:



    savelogs --config=/etc/savelogs.conf



The <B>logB> directive also accepts any standard csh-ish wildcard (e.g., *, ?, [n-m], etc.) for globbing. Globbing is where you specify a wilcard pattern and the argument list is expanded to all filenames that match the pattern. In <B>savelogsB>, this pattern implicitly excludes files whose names end in ’.tar’, ’.gz’, or ’.tgz’ (so you don’t have to worry about compressing already-compressed files).

This is useful if you have log files that are created dynamically and whose names you may not know precisely. For example, say you have a list of files in a directory:



    somelog.010909.gz
    somelog.01090a.gz
    somelog.01090b.gz
    somelog.01090c.gz
    somelog.01090d.gz
    somelog.01090e.gz
    somelog.01090f.gz
    somelog.011001
    somelog.011002
    somelog.011003
    somelog.011004
    somelog.011005
    somelog.011006
    somelog.011007
    somelog.011008
    somelog.011009
    somelog.01100a
    somelog.01100b
    somelog.01100c



To compress these files (the ones that have not already been compressed) you can simply do this:



    savelogs --log=/path/to/somelog.*



The files that end in ’.gz’ are skipped (<B>savelogsB> skips them internally).

Be sure to protect the asterisk (*) with quotes so that the current shell doesn’t try to expand them. You could also do this in a configuration file:



    Log             /path/to/somelog.*



<B>NoLog=/path/to/logB> This is the compliment to the <B>LogB> directive: it removes logs from the list of logs to process. This is useful if you have a log or set of logs that is handled by a separate rotation program or needs special treatment at another time.

For example, if you have many log files listed in your <B>ApacheB> configuration file, you’ll want to take advantage of the <B>ApacheConfB> directive (see below). This will make your <B>savelogsB> configuration file small and easy to understand:



    ## rename and compress all logs found in httpd.conf
    ApacheConf       /www/conf/httpd.conf



This is great, except that there’s this one log that you don’t want <B>savelogsB> to process. Before <B>savelogsB> version 1.40, the only option you had was to list each log individually with the <B>LogB> directive (i.e., you couldn’t use the <B>ApacheConfB> directive at all in such cases). Now, however, you can use the <B>NoLogB> directive to exclude logs that have already been added to the list:



    ## rename and compress all logs found in httpd.conf
    ApacheConf       /www/conf/httpd.conf

    ## ... and exclude joes logs (joe-*_log matches
    ## joe-access_log and joe-error_log)
    NoLog            /www/logs/joe-*_log



You may use full paths or you may use shell wildcard patterns, just like the <B>LogB> directive.

If you have both <B>LogB> and <B>NoLogB> directives, the <B>NoLogB> directive is processed last. This means that:



    Log        /var/log/messages
    NoLog      /var/log/messages



is the same as:



    NoLog      /var/log/messages
    Log        /var/log/messages



and that /var/log/messages will not be processed in either case.

<B>Gripe|NoGripeB> <B>NoGripeB> tells <B>savelogsB> not to complain about not finding any log files to process. By default, <B>savelogsB> gripes about not finding any log files: if you forget to specify any logs (or any directives such as <B>ApacheConfB> that find logs for you) <B>savelogsB> will complain. Also, if you specify log files to process but none of them exist, <B>savelogsB> will similarly complain.

When you turn on <B>NoGripeB>, the complaining is stopped and <B>savelogsB> exits happily.

Example:



    savelogs --nogripe



or in your configuration file:



    Gripe      no



<B>ApacheConf=/path/to/httpd.confB> If you specify this option, giving it a valid httpd.conf file, <B>savelogsB> will parse your Apache configuration file looking for standard log file directives. Any files found will be processed.

Example:



    savelogs --apacheconf=/www/conf/httpd.conf



or in your configuration file:



    ApacheConf          /usr/local/etc/httpd/conf/httpd.conf



Using the <B>ApacheConfB> directive will tell <B>savelogsB> to search through httpd.conf looking for all files associated with <B>TransferLogB>, <B>ErrorLogB>, <B>AgentLogB>, etc. (all those listed in the <B>ApacheLogB> directive) and process them.

<B>ApacheHostB> This option tells <B>savelogsB> which logs to select out of the Apache configuration file (as specified by the <B>ApacheConfB> directive) based on the Apache <B>ServerNameB> directive in the <B>VirtualHostB> block. If this option is set, only logs for matching hosts will be rotated (this applies only to logs found in the Apache configuration file; other logs specified in other ways (e.g., on the command-line or via the <B>LogB> directive) will be processed as usual).

Example:



    savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com



or in your configuration file:



    ApacheConf /www/conf/httpd.conf
    ApacheHost foo.com



The <B>ApacheHostB> directive may be specified multiple times to process logs for multiple virtual hosts. If no logs are found in the Apache <B>VirtualHostB> block, no logs will be rotated for that virtual host.

<B>ApacheLogB> This option allows you to tell <B>savelogsB> which logs to process in the httpd.conf file specified by <B>apacheconfB>. The default value for the <B>apachelogB> directive is:



    TransferLog|ErrorLog|AgentLog|RefererLog|CustomLog



You may do something clever like this:



    savelogs --apacheconf=/www/conf/httpd.conf --apachelog=TransferLog



which would archive all of your access_log files. Then after running this command, you could do this:



    savelogs --process=delete --apacheconf=/www/conf/httpd.conf \
             --apachelog=ErrorLog



which would delete your error_log files.

In general, the fewer values you specify in the <B>apachelogB> directive the faster <B>savelogsB> will find your log files (though the speedup really is negligible, it may also save you from rotating logs you didn’t want to).



    ApacheLog           TransferLog|ErrorLog



would be sufficient for most people using combined Apache logs.

<B>ApacheLogExcludeB> For those who want somewhat finer control of which logs get processed in their Apache configuration file, the <B>apachelogexcludeB> directive allows you to specify a Perl regular expression (which may simply be a string like ’error’) of log files to exclude when processing logs. This way you could do something like this:



    savelogs --apacheconf=/www/conf/httpd.conf --apachelogexclude=logs/bob



which would process all logs found in your httpd.conf file except log files whose names contain the string ’logs/bob’. Maybe Bob likes to rotate his logs using another program or system (conceivable, though unlikely).

Multiple occurances of <B>apachelogexcludeB> are allowed:



    ApacheLogExclude          /dev/null
    ApacheLogExclude          \|
    ApacheLogExclude          logs/bob



which would exclude log files whose names contained ’/dev/null’ or ’logs/bob’ from being processed. Any valid Perl regular expression will work, so:



    ApacheLogExclude          ^/dev/null$



is not the same as the previous example. This example will only match log files whose name is exactly /dev/null, no more, no less.

By default, <B>savelogsB> uses the following patterns to determine logs to exclude:



    ^/dev/null$
    \| (this is a literal pipe character)



This means that by default <B>savelogsB> will not attempt to archive a log whose name is ’/dev/null’ or whose name contains a pipe (|). If for some bizarre reason you wish to remove these defaults when you run <B>savelogsB>, you can give an empty <B>apachelogexcludeB> option on the command-line:



    % savelogs --apachelogexclude= --config=/etc/savelogs.conf



Logs that Apache writes via a pipe must be specified separately using the <B>LogB> directive or on the command-line.

<B>ApacheIncludeB> When specified, this directive tells <B>savelogsB> to read the Apache configuration file (httpd.conf) and follow Inlcude directives (see http://httpd.apache.org/docs/mod/core.html#include for details).

It will work just like Apache does: it will look in directories (when a directory is given as the Include option), it will expand path wildcards (e.g., httpd_[bcd].conf will expand to httpd_b.conf, httpd_c.conf, and httpd_d.conf), and it will work with simple include files as well (e.g., virtual_hosts.conf).

Log files found in Include’ed configuration files will also be processed. <B>savelogsB> has internal consistency checks to ensure that logs are not processed twice, neither are configuration files read twice (thus avoiding those annoying infinite loops).

Example:



    % savelogs --apacheconf=/www/conf/httpd.conf --apacheinclude



or in your configuration file:



    ApacheConf    /www/conf/httpd.conf
    ApacheInclude yes



    Grouping Log File Directives

Beginning with <B>savelogsB> version 1.90, you can apply directives to groups of log files with the <Group> directive in a <B>savelogsB> configuration file (the directive has no command-line equivalent). The following directives apply in a Group block:



  period, ext, sep, datefmt, hourly, touch, chown, chmod, clobber, apachehost, log, disabled



As of version 1.90, the following directives are not honored, but may be at some point in the future:



  size, filter, stem, stemhook, stemlink, nolog, apachelog, apachelogexclude



Group settings override any other settings found in the file, but are applied only to log files found within the Group block.

Example:



    ApacheConf     /www/conf/httpd.conf
    PostMoveHook   apachectl restart
    Period         30

    <Group>
      ApacheHost  www.sample1.tld
      ApacheHost  www.sample2.tld
      Period      10
      Chown       roger:staff
    </Group>

    <Group>
      ApacheHost www.sample3.tld
      ApacheHost www.sample4.tld
      Chown      fonzie
    </Group>



Explanation:

The first Group block will rotate Apache logs for sample1.tld and sample2.tld, but it will only rotate 10 days worth of logs, since the Period directive inside the block is set to 10.

The second Group block will rotate Apache logs for sample3.tld and sample4.tld, and the full 30 day Period setting will apply to them.

Both groups will search /www/conf/httpd.conf and restart with apachectl restart.

Group blocks have a special directive Disabled that when set to true, will skip this block. Griping is also disabled when Disabled is set to true:



  ## this group will be completely skipped.
  <Group>
    Disabled         1
    ApacheHost       www.foo.com
    Chown            roberto
  </Group>



The purpose of the Disabled directive is to prevent ApacheConf from reverting to its behavior in the absence of any ApacheHost directives (i.e., it will parse the entire configuration file and process <B>allB> hosts found). If this is what you desire, then remove or comment out your Group blocks instead of Disabled’ing them.

Known bugs related to the Group directive as of 1.91:

If a log file name found inside of a Group block also matches a NoLog directive outside of the Group block, it will be skipped as if it were outside the block.

If a log referenced inside of a Group block is processed outside of it also (found by some other directive, for example), the first one found will have precedence and the settings inside the Group block may be ignored.

If there are many files being processed, it is possible that in rare circumstances (namely, when processing spans midnight) that one set of logs may be named using one date and another set using another date. This only applies to non-periodic rotation.

    Moving (Renaming) Files

<B>touchB> Touches the original file, creating it if necessary. This is useful for programs that log in append-only mode and do not create the log file if it is missing. Once <B>savelogsB> has renamed a log file, <B>touchB> will create the file if it does not exist, or reset the timestamp if it does.
<B>size=kbytesB> Logs smaller than kbytes will not be included in any processing. To override a default setting, specify <B>--sizeB> with no arguments on <B>savelogsB> command-line.



    Size                      5000



will skip all log files smaller than 5 megs, regardless of other settings.

<B>datefmt=stringB> Allows you to change how dates are formatted using the standard <B>strftimeB> system call. See strftime(1) for format string options. The default string is ’%y%m%d’.



    ## renames logs like this: access_log.02-12-25
    ## Merry Christmas!
    DateFmt                 %y-%m-%d



Some popular options are:



    ## 20020626 (26 June 2002)
    DateFmt                 %Y%m%d

    ## 1.Mar.2002
    DateFmt                 %e.%b.%Y



<B>ext=stringB> Set the filename extension to ’string’. When a file is moved, it is renamed to the original filename plus the extension you specify. If no extension is specified, today’s date is used in ’yymmdd’ format. Options include today and yesterday.

You may not use <B>extB> in a configuration file with a value in backticks (e.g., the line:



    ## this directive will not work: dont use it!
    Ext                  `/bin/date`



in a configuration file will not work). Any other (static) value for <B>extB> in a configuration file will work.

See also <B>hourlyB> below for information on how to modify <B>extB> even further. See <B>datefmtB> above if you want to format your dates differently. <B>extB> is provided chiefly for completeness; its usefulness is limited except in special circumstances where <B>savelogsB> can’t offer a reasonable name for your log.

<B>sep=charB> The separator character to use when moving files. By default this character is a dot (’.’). Other favorites are underscore (’_’) and hyphen/minus sign (’-’).

Example:

Use an underscore character:



    savelogs --sep=_ --process=move /var/log/foo



will rename ~/var/log/foo to ~/var/log/foo_yymmdd (where yymmdd is today’s date).

Use no separator (just concatenate the extension with the filename):



    savelogs --sep= --process=move /var/log/foo



will rename ~/var/log/foo to ~/var/log/fooyymmdd.

<B>hourlyB> Adds a letter of the alphabet to the back of the filename extension. This is useful if you are rotating logs several times a day. For example, if you specified your extension (via <B>extB>) as ’foo’, any log files rotated in the 10am hour will be named ’log.fook’. At 11am, all logs will be called ’log.fool’ and so forth.

Example:



    savelogs /path/to/log_file



This will rename the log file to log_file.yymmdd where yymmdd are today’s year, month, and day of month.



    savelogs --hourly /path/to/log_file



If you specify the <B>hourlyB> option (or in your configuration file, the <B>HourlyB> directive), the log will be renamed to log_file.yymmddz where the z represents the current hour as a letter of the alphabet (0 = a, 1 = b, 2 = c, etc.).

You could also use the <B>datefmtB> directive to get similar results (with much more flexibility to boot). If you rotate more often than once and hour, use the <B>datefmtB> directive or use <B>periodB> to rotate logs with a unique number as the extension. <B>periodB>, discussed below, renames log files with a simple integer: log becomes log.0, the former log.0 becomes log.1 and so forth.

<B>period[=count]B> Renames the file based on a period, which is how frequently you run <B>savelogsB>. If you specify <B>periodB> you may also optionally specify a ’count’, which is how many log files to save using the period option:



    savelogs --period=8 /var/log/messages



You may also use the <B>countB> option, which is deprecated for backward compatibility and some possible future enhancements:



    savelogs --period --count=8 /var/log/messages



If you do not specify a count value (either in <B>periodB> or <B>countB>), a count of 10 is assumed.

The <B>periodB> option will rename the current log to logfile.0, the log that was previously named logfile.0 to logfile.1 and so on, much like <B>B>newsyslog<B>(8)B>.

When you specify the <B>periodB> option, the process phases move and compress are assumed. If you also specify a process phase of filter, that will be honored also.

The <B>periodB> option will override any other <B>sepB> and <B>extB> options specified, using the default dot (’.’) for the separator and an integer for the extension.

The author also recommends you don’t try to mix <B>periodB> named log files with other log files in the same directory, since <B>savelogsB> may not be able to tell which logs are oldest based on the filename extension and destroy the wrong files. You can safely use any default <B>savelogsB> extensions (e.g., the default ’today’ or ’yesterday’ extensions) or your own extension if your own extension contain at least one non-digit (0-9) or your own extension has 5 or more digits. If you meet either of these criteria in your own extension, you may feel confident about mixing logs.

For those familiar with Unix system adminstration, <B>periodB> works like <B>B>newsyslog<B>(8)B> with the <B>BB> option specified in the newsyslog configuration file (the <B>BB> option tells newsyslog to treat logs as binary files and not append the status message to the log).

An example use of the <B>periodB> option:



    savelogs --touch --period=15 /var/log/maillog /var/log/messages



will move any existing ~/var/log/maillog and ~/var/log/messages to ~/var/log/maillog.0 and ~/var/log/messages.0 and compress them. By specifying the <B>touchB> option, the original ~/var/log/maillog and ~/var/log/messages will be ’touched’, recreating the files. When this command is run again, ~/var/log/maillog.0 is moved to ~/var/log/maillog.1 and ~/var/log/maillog is moved to ~/var/log/maillog.0.

<B>countB> Limits the number of logs saved using the <B>periodB> option. The internal default value for <B>countB> is 10.

If you are using the <B>periodB> option, as of version 1.21 you may now simply specify the <B>countB> as part of the <B>periodB> option:



    savelogs --period=5 /var/log/messages



<B>postmovehook=commandB> Runs an arbitrary system command you specify after moving files. If you are rotating Apache log files, you should use a command that will tell your web server to close its log file descriptors and re-open them (e.g., ’restart_apache’).

Example:



    savelogs --apacheconf=/www/conf/httpd.conf \
             --postmovehook=/usr/local/bin/restart_apache



or even nicer in your configuration file:



    PostMoveHook        /usr/local/bin/restart_apache



Paths in <B>postmovehookB> are NOT relative to your home directory, as most other paths are. You should specify the full path to the file if the file is not in your environment’s <B>B>$PATH<B>B>. The exception to this rule is the <B>B>$LOG<B>B> macro which, specified, will automatically be replaced with the current log file path. See <B>VariablesB> below for details.

<B>postmovehookB> is one of two ideal phases to analyze your data before it is archived away (the other phase is the <B>postfilterhookB> phase).

<B>VariablesB>

Some internal <B>savelogsB> variables are available during the postmovehook phase. These variables are automatically interpolated by <B>savelogsB> during execution and are guaranteed to contain some useful value.
<B>B>$APACHE_CONF<B>B> Contains the full path to the Apache configuration file as specified with the <B>apacheconfB> option.



    savelogs --apacheconf=/www/conf/httpd.conf \
             --postmovehook=touch $APACHE_CONF



<B>B>$HOME<B>B> Contains the path to your home directory.



    savelogs --postmovehook=$HOME/bin/myprogram



<B>B>$LOG<B>B> Contains the current log file being processed. If you wanted to run a command on each log file after it is moved, you may enter that command here (on one line). If the line is really long, consider putting it into a shell script that wraps all of your options.



    PostMoveHook    /usr/local/bin/do_something_with_every $LOG



<B>force-pmhB> Executes the <B>postmovehookB> command even if there are no logs to process. By default, <B>savelogsB> will not execute the <B>postmovehookB> command if there are no logs.
<B>chown=uid:gidB> After <B>savelogsB> has renamed your log(s), you may wish to chown them to a new user or group. Use the <B>chownB> option for this. If you don’t specify a user, the user will not change. If you don’t specify a group, the group won’t change.



    ## chown the logs to joe:joegroup
    Chown    joe:joegroup

    ## make the log files owned by the group with gid 500, keep the
    ## current owner
    Chown    :500

    ## change the owner to root, leave the group alone
    Chown    root:



Notice that the colon is necessary at all times.

The <B>chownB> option executes after the <B>PostMoveHookB> phase has completed.

<B>chmodB> After <B>savelogsB> has renamed your log(s), you may wish to change the permissions of the logfile. Use the <B>chmodB> option for this. Permissions should be specified in octal.



    ## make readable only by the owner
    Chmod    0600



The <B>chmodB> option executes after the <B>PostMoveHookB> phase has completed.

<B>stemB> Like <B>extB> except the <B>stemB> is used in addition to <B>extB>. After the move and postmovehook phases have completed, <B>savelogsB> checks to see if you have defined a <B>stemhookB>. If a <B>stemhookB> has been defined, a symbolic link to the log is made using <B>stemB>.

As an example, say you were processing ~/var/log/messages. During the stem phase, <B>savelogsB> would do this:



    messages.today -> messages.<ext>

    (where <ext> is the extension you specified or todays date by
    default)



Once the <B>stemhookB> command has executed, the symbolic link (or hard link or copy, as specified by <B>stemlinkB>) is removed.

The <B>stemB> related options were added in version 1.28.

<B>stemhook=commandB> A command to execute, much like <B>postmovehookB>, except that this phase is suited for log file analysis tools that require a predictably named, dead (i.e., no logging is currently being done to it) log file. <B>urchinB> and <B>analogB> are good examples of programs that require such logs.

An example use for <B>stemhookB> would be:



    % savelogs --stemhook="$HOME/usr/local/urchin/urchin" \
      /www/logs/access_log /www/logs/error_log



<B>urchinB> should be instructed to operate on /www/logs/access_log.today and /www/logs/error_log.today. <B>urchinB> should also be instructed to do nothing to the log files since we’re allowing <B>savelogsB> to manage them for us. Any changes resulting from the <B>stemhookB> command to the log file will occur in the original log file.

The same variables for <B>postmovehookB> are available for <B>stemhookB>.

<B>stemlink=linktypeB> Specify the type of link that should be made during the stem phase. The default is symlink. Other options are hard which creates hard links and copy which creates a copy of the original file. hard links are useful for <B>stemhookB> commands that cannot process symbolic links. If your <B>stemhookB> command modifies the log file, you may wish to choose the copy option which will be discarded after the <B>stemhookB> command is executed.

    Filtering

A filter is simply a program that generates something on STDOUT. They may be pipelines or other programs.
<B>filter=filter_commandB> If the filter process option is specified (via the <B>processB> directive), you should supply a program to filter your log (via the <B>filterB> command), such as egrep or perl (or a pipeline or a shell script containing your commands, etc.).

If no filter command is given, the filter phase will be skipped (even if you specify --process=filter as the <B>processB> directive). Consider following filter command:



    --filter=/usr/bin/egrep -v "/images/" \$LOG



When <B>savelogsB> gets to its filter phase, it will open a pipe to the above command. Output from this command will be saved to a temporary file, then the temporary file will be renamed to replace the original log file.

Notice the strange $LOG variable. This is an internal <B>savelogsB> variable that refers to the location of the log file <B>savelogsB> is currently working on. It is automatically replaced with the right file during execution.

If you are supplying a <B>filterB> command on the command-line, the backslash (\) in front of $LOG is necessary to tell the shell to not interpolate <$LOG> as a shell variable, but instead pass it along untouched to <B>savelogsB>. The backslash is not necessary if you are specifying a <B>filterB> directive in the configuration file:



    Filter          /usr/bin/egrep -v "/images/" $LOG



For the sake of completeness, you can also chain filters via a pipeline, like this:



    Filter    egrep -v "/images/" $LOG | egrep -v "(root|cmd)\.exe" -



The final ’-’ tells egrep to use stdin from the previous pipe for its input.

<B>postfilterhook=commandB> Runs an arbitrary system command you specify after filtering files. See <B>postmovehookB> for examples, including the <B>VariablesB> section.
<B>force-pfhB> Like <B>force-pmhB>, this forces execution of the <B>postfilterhookB> command even if there are no logs (and assuming <B>savelogsB> has reached this phase).

    Archiving and Compressing

<B>gtarB>
<B>tarB> Specifies the location of the <B>tarB> program to use. This defaults to whatever it can find on your system. You usually don’t need to modify this option unless your <B>tarB> or <B>gtarB> program is not in your path.

Example:



    Gtar                 /usr/sbin/gtar



If both <B>gtarB> and <B>tarB> are specified, <B>gtarB> will be used.

<B>archiveB> Specifies the name of the archive to which files will be appended.

This directive is somewhat tricky to understand. Under normal use, <B>savelogsB> uses the name of the file being archived as the archive name. For example, if you were archiving a file named ~/var/log/messages, the name of the archive would be ~/var/log/messages.tar.

If you are archiving multiple files (which is common), each file will be stored in its own archive by name in the directory where the file is located. For example, if you had several files located in ~/var/log, each file would be stored in its own archive named filename.tar in the ~/var/log directory.

If you want to lump together all files in a particular directory into one archive, use the <B>archiveB> directive without any path information:



    savelogs --archive=system.tar \
             /var/log/messages /var/log/proftpd /var/log/foo \
             /www/logs/access_log /www/logs/error_log



This will archive ~/var/log/messages, ~/var/log/proftpd, and ~/var/log/foo in a single file named ~/var/log/system.tar (which may later be compressed if you’ve so specified). ~/www/logs/access_log and ~/www/logs/error_log will also be lumped together in a file called ~/www/logs/system.tar.

If you want to lump together all files for this <B>savelogsB> session into one archive, use the <B>archiveB> directive and specify the full path to the archive:



    savelogs --archive=/var/tmp/logs.tar /var/log/messages \
             --apacheconf=/www/conf/httpd.conf



This will archive ~/var/log/messages and all log files found in the Apache configuration file into a single archive named ~/var/tmp/logs.tar (which may later be compressed).

If you wish to place the archive in your home directory, you may be tempted to just do this:



    savelogs --archive=/logs.tar



This won’t work. Because of the way <B>savelogsB> tries to simplify things relative to your home directory, the leading slash is dropped and <B>savelogsB> doesn’t find any path information (and therefore places the archive in where the files are).

To put files in your home directory, preceed the <B>archiveB> command with a dot-slash:



    savelogs --archive=./logs.tar



This won’t put the archive in your current working directory, as some are wont to assume, but in your home directory. <B>savelogsB> has no notion of a current working directory because it is always changing directories from your home to the directory where the log files are and back.

<B>full-pathB> Specifies whether files stored in archives are full paths relative to your home directory or relative paths relative to the directory in which the file is found.

If <B>full-pathB> is not specified, paths are stored in the tar file relative to their parent directory.

Example:



    savelogs /var/log/messages



will create an archive with the following file in it:



    messages



while



    savelogs --full-path /var/log/messages



will create an archive with the following file in it:



    var/log/messages



When you extract this file later on, the paths will be created for you if they don’t exist, which may not be what you want (but, of course, it may be what you want which is why we have this directive).

<B>gzipB>
<B>compressB>
<B>uncompressB> Specifies the location of the gzip/compress/uncompress binaries for decompressing files (files ending with ’.gz’ or ’.Z’). Use of <B>compressB> and <B>uncompressB> is deprecated. If your system has a <B>gzipB> program in a directory that is not in your $PATH variable, specify its location with this directive. If <B>gzipB> and <B>compressB> are specified, <B>gzipB> will be used.

By default (if none of the above options are specified), <B>savelogsB> will search for a <B>gzipB> binary in your path and use it.

<B>clobberB> If a compressed archive already exists along side a non-compressed archive (e.g., archive.tar and archive.tar.gz), and you’ve instructed <B>savelogsB> to compress archive.tar, some compression programs (like <B>gzipB>) will ask you for confirmation before overwriting existing files.

To get around this, <B>savelogsB> by default enables the ’force’ option on compression programs (usually <B>-fB>). This way, if you’re running <B>savelogsB> from a cron job or another method where there is no controlling terminal, <B>savelogsB> keeps running.

If you’re running <B>savelogsB> interactively (i.e., from a tty) and want <B>savelogsB> to prompt you to overwrite existing compressed files, specify the <B>noclobberB> option:

Example:



    savelogs --noclobber



or in your configuration file:



    Clobber             no



EXAMPLES

The author recommends liberal use of the <B>dry-runB> option when testing these examples or when making big changes to your configuration file or command-line options. Doubly-so when you have the <B>deleteB> process option enabled. There’s no ’undelete’ for UNIX.

    Archiving a single log file



    savelogs /path/to/log_file



By default, <B>savelogsB> will move, and compress a log file. This is its simplest use. If this command is run daily, the result will be a file name like the old file with a yymmdd extension. This file will be compressed.

    Nuking logs

You can use <B>savelogsB> (contrary to its name) to just wipe out log files and reclaim the disk space. If you’ve got a couple of files that from time to time just get too big and there’s really no valuable information in them, do something like this:



    savelogs --process=delete /path/to/log_file1 /path/to/log_file2



If you want to nuke all Apache log files, do something like this:



    savelogs --process=delete --postmovehook=/usr/local/bin/restart_apache \
    --apacheconf=/www/conf/httpd.conf



When you specify only the delete as a process option, no logs are moved, archived, or compressed. They’re just deleted.

    Compressing logs daily

Compressing logs daily is easy:



    savelogs /var/log/messages



will make:



    -rw-r--r--  1 test  vuser  751327 Jul  6 12:48 messages



become:



    -rw-r--r--  1 test  vuser   84625 Jul  6 12:48 messages.010706.gz



    Compressing logs daily if needed



    savelogs --size=5000 /var/log/messages



will only compress the log if the size of the file is 5000 kilobytes (5 megabytes) or larger.

    Using periodic log rotation

You want to save 3 days worth of Apache log files and 5 days worth of system log files. You might try the following lines in your crontab:



    1 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \
    --postmovehook=/usr/local/bin/restart_apache --period=3 \
    --apacheconf=/www/conf/httpd.conf

    5 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \
    --period=5 /var/log/messages /var/log/ftp.log



Most crontab files require that no lines wrap, so you’d need to make sure to keep everything on one line.

Your ~/www/logs directory may look something like this after a week:



    access_log
    access_log.0.gz
    access_log.1.gz
    access_log.2.gz
    error_log
    error_log.0.gz
    error_log.1.gz
    error_log.2.gz



and your system logs directory:



    messages
    messages.0.gz
    messages.1.gz
    messages.2.gz
    messages.3.gz
    messages.4.gz
    ftp.log
    ftp.log.0.gz
    ftp.log.1.gz
    ftp.log.2.gz
    ftp.log.3.gz
    ftp.log.4.gz



    Archiving all logs

Most people want to group their log files in an archive. This makes storing them and retrieving them later for post-processing simple and efficient. Your directory tree might look like this:



    usr/local/etc/httpd/logs
    usr/local/etc/httpd/logs/bar
    usr/local/etc/httpd/logs/bar/access_log
    usr/local/etc/httpd/logs/bar/error_log
    usr/local/etc/httpd/logs/baz
    usr/local/etc/httpd/logs/baz/access_log
    usr/local/etc/httpd/logs/baz/error_log
    usr/local/etc/httpd/logs/biz
    usr/local/etc/httpd/logs/biz/access_log
    usr/local/etc/httpd/logs/biz/error_log
    usr/local/etc/httpd/logs/buz
    usr/local/etc/httpd/logs/buz/access_log
    usr/local/etc/httpd/logs/buz/error_log
    usr/local/etc/httpd/logs/foo
    usr/local/etc/httpd/logs/foo/access_log
    usr/local/etc/httpd/logs/foo/error_log



Issue this command:



    savelogs --process=all --apacheconf=/www/conf/httpd.conf



and your directory tree now looks like this:



    usr/local/etc/httpd/logs
    usr/local/etc/httpd/logs/bar
    usr/local/etc/httpd/logs/bar/access_log.tar.gz
    usr/local/etc/httpd/logs/bar/error_log.tar.gz
    usr/local/etc/httpd/logs/baz
    usr/local/etc/httpd/logs/baz/access_log.tar.gz
    usr/local/etc/httpd/logs/baz/error_log.tar.gz
    usr/local/etc/httpd/logs/biz
    usr/local/etc/httpd/logs/biz/access_log.tar.gz
    usr/local/etc/httpd/logs/biz/error_log.tar.gz
    usr/local/etc/httpd/logs/buz
    usr/local/etc/httpd/logs/buz/access_log.tar.gz
    usr/local/etc/httpd/logs/buz/error_log.tar.gz
    usr/local/etc/httpd/logs/foo
    usr/local/etc/httpd/logs/foo/access_log.tar.gz
    usr/local/etc/httpd/logs/foo/error_log.tar.gz



Inside of each compressed archive is a single file:



    access_log.010523



which is the old log renamed with today’s date. When you run this command again (e.g., from a cron job) tomorrow, you’ll see the same list of files above, except that inside each compressed archive is an additional file:



    access_log.010523
    access_log.010524



    Archiving all logs per directory

Say you want to group together all logs in a single directory in one archive. Your directory tree might look like this:



    usr/local/etc/httpd/logs
    usr/local/etc/httpd/logs/bar
    usr/local/etc/httpd/logs/bar/access_log
    usr/local/etc/httpd/logs/bar/error_log
    usr/local/etc/httpd/logs/baz
    usr/local/etc/httpd/logs/baz/access_log
    usr/local/etc/httpd/logs/baz/error_log
    usr/local/etc/httpd/logs/biz
    usr/local/etc/httpd/logs/biz/access_log
    usr/local/etc/httpd/logs/biz/error_log
    usr/local/etc/httpd/logs/buz
    usr/local/etc/httpd/logs/buz/access_log
    usr/local/etc/httpd/logs/buz/error_log
    usr/local/etc/httpd/logs/foo
    usr/local/etc/httpd/logs/foo/access_log
    usr/local/etc/httpd/logs/foo/error_log



Try this:



    savelogs --process=all --archive=logs.tar --apacheconf=/www/conf/httpd.conf



and your directory tree now looks like this:



    usr/local/etc/httpd/logs
    usr/local/etc/httpd/logs/bar
    usr/local/etc/httpd/logs/bar/logs.tar.gz
    usr/local/etc/httpd/logs/baz
    usr/local/etc/httpd/logs/baz/logs.tar.gz
    usr/local/etc/httpd/logs/biz
    usr/local/etc/httpd/logs/biz/logs.tar.gz
    usr/local/etc/httpd/logs/buz
    usr/local/etc/httpd/logs/buz/logs.tar.gz
    usr/local/etc/httpd/logs/foo
    usr/local/etc/httpd/logs/foo/logs.tar.gz



Inside each ’logs.tar.gz’ are two files:



    access_log.010523
    error_log.010523



    Archiving all logs per session

Say you want to lump all logs in a <B>savelogsB> session into a single archive. Specify the <B>archiveB> option with a full path, like this:



    savelogs --process=all --archive=/tmp/all_logs.tar \
             --apacheconf=/www/conf/httpd.conf /var/log/messages



This will create a single file /tmp/all_logs.tar.gz that contains all logs found in httpd.conf as well as /var/log/messages.

    Filtering logs in place

Maybe your log file fills up with garbage entries that you want to clean out daily. You can use the <B>filterB> option to trim your log in place:



    savelogs --process=filter \
             --filter=/usr/bin/egrep -v "(root|cmd)\.exe" \$LOG \
             --postfilterhook=/usr/local/bin/restart_apache \
             --apacheconf=/www/conf/httpd.conf



This will clean out some Windows worm requests from your Apache log files.

BUGS/CAVEATS

o <B>savelogsB> will not properly interpolate backticks inside the configuration file. Because backticks are shell operators, they must be visible to the shell (meaning they have to be done on the command-line). An example is the <B>extB> command where the command-line option might look like this:



    savelogs --ext=`smalldate -m`



there is no corresponding configuration file entry.

o You must obey shell escaping rules when you’re doing <B>filterB> commands on the command-line. Sometimes these can be tricky. For example, double quotes will do different things than single quotes (I recommend single quotes because you can then use the <B>B>$LOG<B>B> variable without big headaches). Using a config file sometimes helps.
o You can’t have multiline commands in the configuration file. This is on the to do list.

WISHLIST/TODO

o Would be nice to use the VirtualHost ’User’ directive for the <B>chownB> option. Maybe a ’ApacheChown’ option.
o Optimize the ApacheHost directive to jump out of the Apache config parsing once all hosts have been found (can you have a VirtualHost block appear twice with the same ServerName directive?)

ACKNOWLEDGEMENTS

o Thanks to Jeroen Latour (cpantester@calaquendi.net) for working with me on getting all the tests to run cleanly on his Cobalt box. It should now test cleanly on many other platforms because of his patience.

AUTHOR

Scott Wiersdorf, <scott@perlcode.org>

COPYRIGHT

Copyright (c) 2001-2004 Scott Wiersdorf. All rights reserved.

POD ERRORS

Hey! <B>The above document had some coding errors, which are explained below:B>
Around line 2465: =back doesn’t take any parameters, but you said =back 4
Search for    or go to Top of page |  Section 1 |  Main Index


perl v5.20.3 SAVELOGS (1) 2011-10-15

Powered by GSP Visit the GSP FreeBSD Man Page Interface.
Output converted with manServer 1.07.