GSP
Quick Navigator

Search Site

Unix VPS
A - Starter
B - Basic
C - Preferred
D - Commercial
MPS - Dedicated
Previous VPSs
* Sign Up! *

Support
Contact Us
Online Help
Handbooks
Domain Status
Man Pages

FAQ
Virtual Servers
Pricing
Billing
Technical

Network
Facilities
Connectivity
Topology Map

Miscellaneous
Server Agreement
Year 2038
Credits
 

USA Flag

 

 

Man Pages


Manual Reference Pages  -  BRACKUP::INVENTORYDATABASE (3)

.ds Aq ’

NAME

Brackup::InventoryDatabase - track what chunks are already on a target

CONTENTS

DESCRIPTION

The Brackup InventoryDatabase keeps track of which chunks (files) are already on a given target, speeding up future iterative backups, negating the need to ask the target for each chunk whether it exists or not (which may be a lot of network roundtrips, slow even in the best of network conditions).

Unlike the Digest Cache, the inventory database is not a cache... its contents matter. Consider what happens when the inventory database doesn’t match reality:
<B>1) Exists in inventory database; not on targetB> If a chunk exists in the inventory database, but not on the target, brackup won’t store it on the target, and you’ll think a backup succeeded, but it’s not actually there.
<B>2a) Exists on target; not in inventory database (without encryption)B> You re-upload it to the target, so you waste time & bandwidth, but no extra disk space is wasted, and no chunks are orphaned. Actually, chunks are un-orphaned, as the inventory database is now updated and contains the chunk you just uploaded.
<B>2b) Exists on target; not in inventory database (with encryption)B> When using encryption, each time a chunk is encrypted with gpg, the contents are different. So if the inventory database says a given chunk isn’t already stored on the server, it will be re-encrypted and stored (uploaded) again. You may or may not have an orphaned chunk on the server, depending on whether or not it’s referenced by any other *.brackup meta files.
For those reasons, it’s somewhat important that your inventory database be kept around and not deleted. If you’re running brackup to the same target from different computers, you might want to sync up your inventory databases with each other, so you don’t do unnecessary uploads to the target.

Tools to rebuild your inventory database from the target’s enumeration of its chunks and the target’s *.brackup metafiles isn’t yet done, but would be pretty easy. (this is a TODO item)

In any case, it’s not tragic if you lose your inventory database... it just means you’ll need to upload more stuff and maybe waste some disk space until you next run a ’brackup-target gc’ garbage collection, which cleans up orphaned chunks. If you’re feeling paranoid, it’s safer to delete your inventory database, tricking Brackup into thinking your target is empty (even if it’s not), rather than Brackup thinking your target has something when it actually doesn’t.

DETAILS

    Storage type

The inventory database makes use of Dictionary modules (Brackup::Dict::*) to handle database storage. The default dictionary used is Brackup::Dict::SQLite, which stores the cache as an SQLite database in a single file. The schema is created automatically as needed... no database maintenance is required.

The dictionary type can be specified in the [TARGET] declaration in your brackup.conf file, using the ’inventorydb_type’ property e.g.:



  [TARGET:amazon]
  type = Amazon
  aws_access_key_id  = ...
  aws_secret_access_key =  ...
  # specify the lighter/slower Brackup::Dict::SQLite2 instead of the default
  inventorydb_type = SQLite2



    File location

The inventory database file (for file-based dictionaries) is stored in either the location specified in the ’inventorydb_file’ property of a [TARGET] declaration in ~/.brackup.conf e.g.:



  [TARGET:amazon]
  type = Amazon
  aws_access_key_id  = ...
  aws_secret_access_key =  ...
  inventorydb_file = /home/bradfitz/.amazon-already-has-these-chunks.db



Or, more commonly (and recommended), is to not specify it and accept the default location, which is .brackup-target-TARGETNAME.invdb in your home directory (where it might be shared by multiple backup roots).

    SQLite Schema

This is made automatically for you, but if you want to look around in it, the schema is:



  CREATE TABLE target_inv (
       key TEXT PRIMARY KEY,
       value TEXT
  )



    Keys & Values stored in the database

<B>KeysB>

The key is the digest of the raw (pre-compression/encryption) file/chunk (with GPG recipient, if using encryption), and the value is the digest of the chunk stored on the target, which contains the raw chunk. The chunk stored on the target may contain other chunks, may be compressed, encrypted, etc.



 <raw_digest>               --> <stored_digest> <stored_length>
 <raw_digest>;to=<gpg_rcpt> --> <stored_digest> <stored_length>



For example:



  sha1:e23c4b5f685e046e7cc50e30e378ab11391e528e;to=6BAFF35F =>
     sha1:d7257184899c9e6c4e26506f1c46f8b6562d9ee7 71223



Means that the chunk with sha1 contents e23c4..., intended to be en/de-crypted for 6BAFF35F, can be got by asking the target for the chunk with digest d72571848..., with length 71,223 bytes.

When using the Brackup feature which combines small files into larger blobs, the inventory database instead stores values like:



  <raw_digest>[;to=<gpg_rcpt>] -->
     <stored_digest> <stored_length> <from_offset>-<to_offset>



Which is the same thing, but after fetching the composite chunk using the stored digest provided, only the range provided from from_offset to to_offset should be used.

SEE ALSO

Brackup

Brackup::Target

Search for    or go to Top of page |  Section 3 |  Main Index


perl v5.20.3 BRACKUP::INVENTORYDATABASE (3) 2010-03-27

Powered by GSP Visit the GSP FreeBSD Man Page Interface.
Output converted with manServer 1.07.