|1.||Crawl all directories, and print all the names found to standard output. Also check for ._rdup_. files to restore the original ownership (see the -u flag).|
Write a filelist with all the names found when crawling.
Use this list to calculate the correct incremental dump.
And for incremental dumps
|1.||Read in the filelist that was written when doing a full dump.|
|2.||Touch the time stamp file.|
|3.||Crawl all the directories again.|
|4.||Diff 1. and 2. to get two lists; one of removed items and one of added/modified items.|
|5.||Write the removed items to standard output|
|6.||Write the modified/new items to standard output.|
Write a new filelist.
The DIRS/FILES can be specified multiple times. These are the directories and files you want to backup. If omitted it defaults to the current directory "." .
If the -N timestamp option is not given, all paths found are printed. Only when a -N timestamp file is given, times can be compared and an incremental output can be generated.
rdup prints a filelist to standard output. Subsequent programs in a pipe line can be used to actually implement to backup scheme. After a run a new FILELIST is written. No warning is given when FILELIST is an existing file, it just gets overwritten by rdup. New runs will print out only those files that have actually changed or are removed since the last run, thereby making incremental backups possible.
Files are checked for changes by comparing the c-time (change time), if this time is NEWER than the c-time of timestamp file the pathname is printed to standard output. When files are removed they are also printed to standard output, but they are prefixed with a -. See FORMAT below. The default format rdup uses is: "%p%T %b %t %u %U %g %G %l %s\n%n%C"
Note, that rdup also supports hashing of files, this makes it possible to check the local hash with the hash of the backed up file.
All errors are written to standard error. If the directory or file does not exist, they are skipped and a warning is emitted.
The general idea is to be very UNIX like and create a bunch of simple programs which each do a their specific thing very well. With rdup and a small shell script (50 lines) one can implement encrypted and compressed backups.
As rdup doesnt backup anything, the backup policy; what you backup, how you backup, how often and how you restore; is all left to the scripts and your imagination. To kick start your imagination see rdup-tr(1), rdup-up(1) and maybe rdup-backups.
-Pcommand,opt0,...,opt6 Filter all output through command. opt0 through opt6 are given as options to the command. Multiple -Ps can be used, there is however a maximum of seven options for each command. The options are separated with commas, there must be no space in between.
Due to the nature of pipes in Unix, this pipeline is recreated for every file processed. Also see Child Processes below.
-F format Specify a printf-style format to use. See FORMAT below. -N timestamp use the c_time of file timestamp as the timestamp to decide what to include in the incremental backup list. If timestamp does not exist a full dump is performed. -M timestamp As -N, but look at the m_time of timestamp. -R Reverse the output of rdup. Tools accepting this ouput must create leading directory as they see them. This option allows a script -- running as a normal user -- to put files in a directory which could have 0600 as its permission. -E file The file named file contains a list of Perl-compatible regular expressions (PCRE), one per line, that rdup will use to exclude names. A # at the start of the line can be used to signal a comment. Empty lines are ignored.
If a directory is excluded, rdup wont descend in that directory, so all files in that directory are also excluded.
The directories leading up to the directory to be backed up can not be excluded. If you use a command line like:
rdup /dev/null /home/miekg/bin
The directories /home, /home/miekg, /home/miekg/bin are always printed.
If you want to exclude the file /home/miekg/blaat you need to add the following regular expression: /home/miekg/blaat.
If you want to exclude all .mozilla/cache directories of all users you can use /home/.*/.mozilla/cache/.*. This doesnt exclude the directory itself and Im assuming that the users home directories are found under /home.
Also note that rdup does not print directories with a trailing slash.
-a Restore the original access times on files and directories. -n Dont honor .nobackup files. Normally if such a file is found the directory and all files containing it, are not printed to standard output. Now they are. -u Output files with a ._rdup_. prefix just as normal files. When this option is not given these files are skipped by rdup. -r Only print removed files; entries that start with a -. This option unsets -m. -m Only print modified/new files; entries that start with a +. This option unsets -r. -v Be more verbose. When used each path will also be printed to standard error. -s size Dont output files larger than size bytes. This can be used to limit the amount of data to be transferred when doing a remote backup. This option only applies to files. -x Stay on the local filesystem. -V Print rdups version. -h Give an overview of the options.
When creating output you might also want to pipe the contents of each file through a number of commands, say a compression and encryption utility. Note that this is different than compressing the entire archive as GNU tar allows by using the -z option. So this is where rdup comes in. It allows you to create a normal archive in which each file is encrypted (or compressed. reversed or whatever). rdup does this by forking child processes which transform the content.
If one of the forked children returns an exit code other than zero (0), it is assumed the whole conversion process failed. In that case rdup terminates.
As said rdup works by forking off a number of child processes (those commands named with the -P option(s)), interconnecting these with pipes. The current file is connected to the first child. The output created by these child processes is captured by the parent (rdup). The contents is then written to standard output in an archive format. As a picture says more than a thousand words here is an ASCII image of the process:
+--- ... (stdout) ... ----> archive
rdup <--- ... ... <----+
loop #files |
file ---> cmd1 | cmd2 | ...| cmdN
rm -f timestamp && rdup -N timestamp LIST DIR
A full-dump filelist is printed to standard output. And with:
rdup -N timestamp LIST DIR
An incremental dump filelist is printed. The file timestamp is used to save the exact time of rdups run. The file LIST is used to calculate the correct incremental dump list, this is needed for files that are removed, or have a different type.
If backups are made by a non-root user or on a filesystem that does not implement/allow chown (think sshfs mounted by an ordinary user), rdup creates a seperate file which stores the correct user and group information. If the files name is foobar a new file called ._rdup_.foobar is created in the same directory. This file contains one line, for instance:
That tells that the actual ownership should be root:root. For directories a ._rdup_. file is created inside the current directory.
The default format rdup uses is: "%p%T %b %t %u %U %g %G %l %s\n%n%C"
The following escape sequences are understood by rdup:
p: + if file is new/modified, - if removed
b: permission bits from lstat(2), octal in four digits
m: the file mode bits, st_mode from lstat(2), decimal digits
l: path name length
s: original file size, but see CAVEATS
n: path name
N: path name, but in case of a soft- or hardlink only the link name.
t: time of modification (seconds from epoch)
H: the SHA1 hash of the regular file, all zeros ("0") for all other types
T: file type
- normal file, l symlink, h hardlink, d directory,
c character device, b block device, p named pipe
and s socket.
C: the content of the file (none for all other types)
To delimit the output of rdup with NULLs you can use \0 in the format string.
Any file content is written in a block/chunk based manner. The last block is signaled with a null block. A block start entry is ASCII and is formatted as follows: VVBLOCKBBBBB\n . Where VV is the version, currently at 01, then the literal string BLOCK and then the amount of bytes (BBBBB), typical 08192. And then a newline. This look like this:
<START OF THE FIRST 8192 BYTES>01BLOCK00015
<ANOTHER 15 BYTES>01BLOCK00000
A byte count of zero signals a stop block.
rdup writes the (internal) FILELIST in the following format:
MODE DEV INODE LINK UID GID PATH_SIZE FILE_SIZE PATH
Where MODE is the st_mode from stat(2), DEV is the dev id as returned by the stat call and INODE is the inode number - rdup needs this info to decide if a directory is renamed. LINK is equal to h for hardlinks, l for symlinks and otherwise it is *. UID and GID are the numeric user and group id of the file. PATH_SIZE is the length of PATH. FILE_SIZE the file size. And finally PATH is the path of the file.
A typical example is:
16893 2050 32085 * 1000 1000 30 4096 /home/miekg/git/rdup/.git/logs
The output generated by rdup is formatted like:
+|-TYPE BITS MTIME UID USER GID GROUP PATH_SIZE FILE_SIZE\n
This makes it possible possible for a remote shell script to receive the actual file contetns and make a backup.
For directories: the FILE_SIZE is zero and no content is printed. Thus:
+d 0755 1260243445 1000 miekg 1000 miekg 11 0\n
For regular files the following is a sample output:
+- 0644 1260243445 1000 miekg 1000 miekg 32 6\n
Where aaa/a is a regular file containing the word hello\n
Soft- and hardlinks are handled differently when using %n, if you dont like this behavior use %N. The PATH name is generated from the links name and its target. A symlink like
/home/bin/blaat -> /home/bin/bliep
is printed as /home/bin/blaat -> /home/bin/bliep. The PATH_SIZE is modified accordingly, where -> (4 characters) is also counted. The FILE_SIZE is not needed for soft- or hardlinks, so it is set the length of the links name -- the part left of the ->, in this case the length of /home/bin/blaat.
If rdup encounters a hardlink it is handled in the same way, but the output type is set to h instead of l. A hardlink is only detected if rdup finds a file with the same inode and device number as a previous one, i.e. such hardlinks must be contained in your backup.
Again note: with %N only the links name is printed. The FILE_SIZE is still set to the length of the links name.
For devices the size field (%s) is changed to hold the major,minor number of the device. So if a major number is 8 and the minor number is 0 (under Linux this is /dev/sda), its size will be 8,0. The numbers are only separated with a comma ,.
You will probably think rdup will descend into the directory the symbolic link points to. This is not what actually happens, rdup will print any directories leading up to the symlink and will not descend into the directory. GNU tar works the same.
rdup return a zero exit code on success, otherwise 1 is returned. rdup will abort if a file can not be concatenated, if a regular expression can not be compiled or if a signal is received.
The next set of examples will all make a full dump -- because of the use of /dev/null. See rdup-tr(1) for more advanced examples.
rdup /dev/null ~/bin | rdup-up -t /shared/backuprdup /dev/null /shared/backup | rdup-up -t /tmp/restoreorcp -rap /shared/backup /tmp/restore
Backup:rdup /dev/null ~/bin > my-archive.rdupRestore:rdup-up -t /tmp/restore < my-archive.rdup
Backup:rdup -R -F %N\n /dev/null ~/bin | cpio -o -Hcrc > my-archive.cpioRestore:cpio -i -d -Hcrc < my-archive.cpio
Backup:rdup -F %N\n /dev/null ~/bin | tar c -f my-archive.tar -T - --no-recursionRestore:tar x -f my-archive.tar
Written by Miek Gieben.
Report bugs to <email@example.com>.
http:/www.miek.nl/projects/rdup is the main site of rdup. Also see rdup-tr(1), rdup-up(1) and rdup-backups(7).
Copyright (C) 2005-2011 Miek Gieben. This is free software. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Licensed under the GPL version 3. See the file LICENSE in the source distribution of rdup.
|1.1.14||RDUP (1)||24 Dec 2005|