NAME

TokyoTyrant - Pure Perl Interface of Tokyo Tyrant

SYNOPSYS

 use TokyoTyrant;

INTRODUCTION

This module implements the pure Perl client which connects to the server of Tokyo Tyrant and speaks its original binary protocol.

Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet. Though the DBM has high performance, you might bother in case that multiple processes share the same database, or remote processes access the database. Thus, Tokyo Tyrant is provided for concurrent and remote connections to Tokyo Cabinet. It is composed of the server process managing a database and its access library for client applications.

The server features high concurrency due to thread-pool modeled implementation and the epoll/kqueue mechanism of the modern Linux/*BSD kernel. The server and its clients communicate with each other by simple binary protocol on TCP/IP. Protocols compatible with memcached and HTTP/1.1 are also supported so that almost all principal platforms and programming languages can use Tokyo Tyrant. High availability and high integrity are also featured due to such mechanisms as hot backup, update logging, and replication. The server can embed Lua, a lightweight script language so that you can define arbitrary operations of the database.

Because the server uses the abstract API of Tokyo Cabinet, all of the six APIs: the on-memory hash database API, the on-memory tree database API, the hash API, the B+ tree database API, the fixed-length database API, and the table database API, are available from the client with the common interface. Moreover, the table extension is provided to use specifidc features of the table database.

Setting

Get this package and extract it.

Enter the directory of the extracted package then perform installation.

 perl Makefile.PL
 make
 su
 make install

The package `TokyoTyrant' should be loaded in each source file of application programs.

 use TokyoTyrant;

If you want to enable runtime assertion, set the variable `$TokyoTyrant::DEBUG' to be true.

 $TokyoTyrant::DEBUG = 1;

EXAMPLE

The following code is an example to use a remote database.

 use TokyoTyrant;
 use strict;
 use warnings;
 
 # create the object
 my $rdb = TokyoTyrant::RDB->new();
 
 # connect to the server
 if(!$rdb->open("localhost", 1978)){
     my $ecode = $rdb->ecode();
     printf STDERR ("open error: %s\n", $rdb->errmsg($ecode));
 }
 
 # store records
 if(!$rdb->put("foo", "hop") ||
    !$rdb->put("bar", "step") ||
    !$rdb->put("baz", "jump")){
     my $ecode = $rdb->ecode();
     printf STDERR ("put error: %s\n", $rdb->errmsg($ecode));
 }
 
 # retrieve records
 my $value = $rdb->get("foo");
 if(defined($value)){
     printf("%s\n", $value);
 } else {
     my $ecode = $rdb->ecode();
     printf STDERR ("get error: %s\n", $rdb->errmsg($ecode));
 }
 
 # traverse records
 $rdb->iterinit();
 while(defined(my $key = $rdb->iternext())){
     my $value = $rdb->get($key);
     if(defined($value)){
         printf("%s:%s\n", $key, $value);
     }
 }
 
 # close the connection
 if(!$rdb->close()){
     my $ecode = $rdb->ecode();
     printf STDERR ("close error: %s\n", $rdb->errmsg($ecode));
 }
 
 # tying usage
 my %hash;
 if(!tie(%hash, "TokyoTyrant::RDB", "localhost", 1978)){
     printf STDERR ("tie error\n");
 }
 $hash{"quux"} = "touchdown";
 printf("%s\n", $hash{"quux"});
 while(my ($key, $value) = each(%hash)){
     printf("%s:%s\n", $key, $value);
 }
 untie(%hash);

The following code is an example to use a remote database with the table extension.

 use TokyoTyrant;
 use strict;
 use warnings;
 
 # create the object
 my $rdb = TokyoTyrant::RDBTBL->new();
 
 # connect to the server
 if(!$rdb->open("localhost", 1978)){
     my $ecode = $rdb->ecode();
     printf STDERR ("open error: %s\n", $rdb->errmsg($ecode));
 }
 
 # store a record
 my $pkey = $rdb->genuid();
 my $cols = { "name" => "mikio", "age" => "30", "lang" => "ja,en,c" };
 if(!$rdb->put($pkey, $cols)){
     my $ecode = $rdb->ecode();
     printf STDERR ("put error: %s\n", $rdb->errmsg($ecode));
 }
 
 # store another record
 $cols = { "name" => "falcon", "age" => "31", "lang" => "ja", "skill" => "cook,blog" };
 if(!$rdb->put("x12345", $cols)){
     my $ecode = $rdb->ecode();
     printf STDERR ("put error: %s\n", $rdb->errmsg($ecode));
 }
 
 # search for records
 my $qry = TokyoTyrant::RDBQRY->new($rdb);
 $qry->addcond("age", $qry->QCNUMGE, "20");
 $qry->addcond("lang", $qry->QCSTROR, "ja,en");
 $qry->setorder("name", $qry->QOSTRASC);
 $qry->setlimit(10);
 my $res = $qry->search();
 foreach my $rkey (@$res){
     my $rcols = $rdb->get($rkey);
     printf("name:%s\n", $rcols->{name});
 }
 
 # close the connection
 if(!$rdb->close()){
     my $ecode = $rdb->ecode();
     printf STDERR ("close error: %s\n", $rdb->errmsg($ecode));
 }

DESCRIPTION

Class TokyoTyrant::RDB

Remote database is a set of interfaces to use an abstract database of Tokyo Cabinet, mediated by a server of Tokyo Tyrant. Before operations to store or retrieve records, it is necessary to connect the remote database object to the server. The method `open' is used to open a database connection and the method `close' is used to close the connection.

$rdb = TokyoTyrant::RDB->new()

Create a remote database object.

The return value is the new remote database object.

$rdb->errmsg(ecode)

Get the message string corresponding to an error code.

`ecode' specifies the error code. If it is not defined or negative, the last happened error code is specified.

The return value is the message string of the error code.

$rdb->ecode()

Get the last happened error code.

The return value is the last happened error code.

The following error code is defined: `$rdb->ESUCCESS' for success, `$rdb->EINVALID' for invalid operation, `$rdb->ENOHOST' for host not found, `$rdb->EREFUSED' for connection refused, `$rdb->ESEND' for send error, `$rdb->ERECV' for recv error, `$rdb->EKEEP' for existing record, `$rdb->ENOREC' for no record found, `$rdb->EMISC' for miscellaneous error.

$rdb->open(host, port, timeout)

Open a remote database connection.

`host' specifies the name or the address of the server.

`port' specifies the port number. If it is not defined or not more than 0, UNIX domain socket is used and the path of the socket file is specified by the host parameter.

`timeout' specifies the timeout of each query in seconds. If it is not defined or not more than 0, the timeout is not specified.

If successful, the return value is true, else, it is false.

$rdb->close()

Close the database connection.

If successful, the return value is true, else, it is false.

$rdb->put(key, value)

Store a record.

`key' specifies the key.

`value' specifies the value.

If successful, the return value is true, else, it is false.

If a record with the same key exists in the database, it is overwritten.

$rdb->putkeep(key, value)

Store a new record.

`key' specifies the key.

`value' specifies the value.

If successful, the return value is true, else, it is false.

If a record with the same key exists in the database, this method has no effect.

$rdb->putcat(key, value)

Concatenate a value at the end of the existing record.

`key' specifies the key.

`value' specifies the value.

If successful, the return value is true, else, it is false.

If there is no corresponding record, a new record is created.

$rdb->putshl(key, value, width)

Concatenate a value at the end of the existing record and shift it to the left.

`key' specifies the key.

`value' specifies the value.

`width' specifies the width of the record.

If successful, the return value is true, else, it is false.

If there is no corresponding record, a new record is created.

$rdb->putnr(key, value)

Store a record without response from the server.

`key' specifies the key.

`value' specifies the value.

If successful, the return value is true, else, it is false.

If a record with the same key exists in the database, it is overwritten.

$rdb->out(key)

Remove a record.

`key' specifies the key.

If successful, the return value is true, else, it is false.

$rdb->get(key)

Retrieve a record.

`key' specifies the key.

If successful, the return value is the value of the corresponding record. `undef' is returned if no record corresponds.

$rdb->mget(recs)

Retrieve records.

`recs' specifies the reference to a hash containing the retrieval keys. As a result of this method, keys existing in the database have the corresponding values and keys not existing in the database are removed.

If successful, the return value is the number of retrieved records or -1 on failure.

$rdb->vsiz(key)

Get the size of the value of a record.

`key' specifies the key.

If successful, the return value is the size of the value of the corresponding record, else, it is -1.

$rdb->iterinit()

Initialize the iterator.

If successful, the return value is true, else, it is false.

The iterator is used in order to access the key of every record stored in a database.

$rdb->iternext()

Get the next key of the iterator.

If successful, the return value is the next key, else, it is `undef'. `undef' is returned when no record is to be get out of the iterator.

It is possible to access every record by iteration of calling this method. It is allowed to update or remove records whose keys are fetched while the iteration. However, it is not assured if updating the database is occurred while the iteration. Besides, the order of this traversal access method is arbitrary, so it is not assured that the order of storing matches the one of the traversal access.

$rdb->fwmkeys(prefix, max)

Get forward matching keys.

`prefix' specifies the prefix of the corresponding keys.

`max' specifies the maximum number of keys to be fetched. If it is not defined or negative, no limit is specified.

The return value is the reference to an array of the keys of the corresponding records. This method does never fail. It returns an empty array even if no record corresponds.

Note that this method may be very slow because every key in the database is scanned.

$rdb->addint(key, num)

Add an integer to a record.

`key' specifies the key.

`num' specifies the additional value. If it is not defined, 0 is specified.

If successful, the return value is the summation value, else, it is `undef'.

If the corresponding record exists, the value is treated as an integer and is added to. If no record corresponds, a new record of the additional value is stored. Because records are stored in binary format, they should be processed with the `unpack' function with the `i' operator after retrieval.

$rdb->adddouble(key, num)

Add a real number to a record.

`key' specifies the key.

`num' specifies the additional value. If it is not defined, 0 is specified.

If successful, the return value is the summation value, else, it is `undef'.

If the corresponding record exists, the value is treated as a real number and is added to. If no record corresponds, a new record of the additional value is stored. Because records are stored in binary format, they should be processed with the `unpack' function with the `d' operator after retrieval.

$rdb->ext(name, key, value, opts)

Call a function of the script language extension.

`name' specifies the function name..

`key' specifies the key. If it is not defined, an empty string is specified.

`value' specifies the value. If it is not defined, an empty string is specified.

`opts' specifies options by bitwise-or: `$rdb->XOLCKREC' for record locking, `$rdb->XOLCKGLB' for global locking. If it is not defined, no option is specified.

If successful, the return value is the value of the response or `undef' on failure.

$rdb->sync()

Synchronize updated contents with the file and the device.

If successful, the return value is true, else, it is false.

$rdb->optimize(params)

Optimize the storage.

`params' specifies the string of the tuning parameters. If it is not defined, it is not used.

If successful, the return value is true, else, it is false.

$rdb->vanish()

Remove all records.

If successful, the return value is true, else, it is false.

$rdb->copy(path)

Copy the database file.

`path' specifies the path of the destination file. If it begins with `@', the trailing substring is executed as a command line.

If successful, the return value is true, else, it is false. False is returned if the executed command returns non-zero code.

The database file is assured to be kept synchronized and not modified while the copying or executing operation is in progress. So, this method is useful to create a backup file of the database file.

$rdb->rnum()

Get the number of records.

The return value is the number of records or 0 if the object does not connect to any database server.

$rdb->size()

Get the size of the database.

The return value is the size of the database or 0 if the object does not connect to any database server.

$rdb->stat()

Get the status string of the database server.

The return value is the status message of the database or `undef' if the object does not connect to any database server. The message format is TSV. The first field of each line means the parameter name and the second field means the value.

$rdb->misc(name, args, opts)

Call a versatile function for miscellaneous operations.

`name' specifies the name of the function. All databases support "putlist", "outlist", and "getlist". "putlist" is to store records. It receives keys and values one after the other, and returns an empty list. "outlist" is to remove records. It receives keys, and returns an empty array. "getlist" is to retrieve records. It receives keys, and returns keys and values of corresponding records one after the other. Table database supports "setindex", "search", and "genuid".

`args' specifies the reference to an array of arguments. If it is not defined, no argument is specified.

`opts' specifies options by bitwise-or: `$rdb->MONOULOG' for omission of the update log. If it is not defined, no option is specified.

If successful, the return value is the reference to an array of the result. `undef' is returned on failure.

Tying functions of TokyoTyrant::RDB

tie(%hash, "TokyoTyrant::RDB", host, port)

Tie a hash variable to a remote database file.

`host' specifies the name or the address of the server.

`port' specifies the port number. If it is not defined or not more than 0, UNIX domain socket is used and the path of the socket file is specified by the host parameter.

If successful, the return value is true, else, it is false.

untie(%hash)

Untie a hash variable from the database connection.

The return value is always true.

$hash{key} = value

Store a record.

`key' specifies the key.

`value' specifies the value.

If successful, the return value is true, else, it is false.

If a record with the same key exists in the database, it is overwritten.

delete($hash{key})

Remove a record.

`key' specifies the key.

If successful, the return value is true, else, it is false.

$hash{key}

Retrieve a record.

`key' specifies the key.

If successful, the return value is the value of the corresponding record. `undef' is returned if no record corresponds.

exists($hash{key})

Check whether a record corrsponding a key exists.

`key' specifies the key.

The return value is true if the record exists, else it is false.

$hash = ()

Remove all records.

The return value is always `undef'.

(the iterator)

The inner methods `FIRSTKEY' and `NEXTKEY' are also implemented so that you can use the tying functions `each', `keys', and so on.

Class TokyoTyrant::RDBTBL

This class inherits the class "TokyoTyrant::RDB". All methods are specific to servers of the table database.

$rdb->put(pkey, cols)

Store a record.

`pkey' specifies the primary key.

`cols' specifies the reference to a hash containing columns.

If successful, the return value is true, else, it is false.

If a record with the same key exists in the database, it is overwritten.

$rdb->putkeep(pkey, cols)

Store a new record.

`pkey' specifies the primary key.

`cols' specifies the reference to a hash containing columns.

If successful, the return value is true, else, it is false.

If a record with the same key exists in the database, this method has no effect.

$rdb->putcat(pkey, cols)

Concatenate columns of the existing record.

`pkey' specifies the primary key.

`cols' specifies the reference to a hash containing columns.

If successful, the return value is true, else, it is false.

If there is no corresponding record, a new record is created.

$rdb->out(pkey)

Remove a record.

`pkey' specifies the primary key.

If successful, the return value is true, else, it is false.

$rdb->get(pkey)

Retrieve a record.

`pkey' specifies the primary key.

If successful, the return value is the reference to a hash of the columns of the corresponding record. `undef' is returned if no record corresponds.

$rdb->mget(recs)

Retrieve records.

`recs' specifies the reference to a hash containing the retrieval keys. As a result of this method, keys existing in the database have the corresponding columns and keys not existing in the database are removed.

If successful, the return value is the number of retrieved records or -1 on failure.

Due to the protocol restriction, this method can not handle records with binary columns including the "\0" chracter.

$rdb->setindex(name, type)

Set a column index.

`name' specifies the name of a column. If the name of an existing index is specified, the index is rebuilt. An empty string means the primary key.

`type' specifies the index type: `$rdb->ITLEXICAL' for lexical string, `$rdb->ITDECIMAL' for decimal string, `$rdb->ITTOKEN' for token inverted index, `$rdb->ITQGRAM' for q-gram inverted index. If it is `$rdb->ITOPT', the index is optimized. If it is `$rdb->ITVOID', the index is removed. If `$rdb->ITKEEP' is added by bitwise-or and the index exists, this method merely returns failure.

If successful, the return value is true, else, it is false.

$rdb->genuid()

Generate a unique ID number.

The return value is the new unique ID number or -1 on failure.

Class TokyoTyrant::RDBQRY

This class is a helper for the class "TokyoTyrant::RDBTBL".

$qry = TokyoTyrant::RDBQRY->new(rdb)

Create a query object.

`rdb' specifies the remote database object.

The return value is the new query object.

$qry->addcond(name, op, expr)

Add a narrowing condition.

`name' specifies the name of a column. An empty string means the primary key.

`op' specifies an operation type: `$qry->QCSTREQ' for string which is equal to the expression, `$qry->QCSTRINC' for string which is included in the expression, `$qry->QCSTRBW' for string which begins with the expression, `$qry->QCSTREW' for string which ends with the expression, `$qry->QCSTRAND' for string which includes all tokens in the expression, `$qry->QCSTROR' for string which includes at least one token in the expression, `$qry->QCSTROREQ' for string which is equal to at least one token in the expression, `$qry->QCSTRRX' for string which matches regular expressions of the expression, `$qry->QCNUMEQ' for number which is equal to the expression, `$qry->QCNUMGT' for number which is greater than the expression, `$qry->QCNUMGE' for number which is greater than or equal to the expression, `$qry->QCNUMLT' for number which is less than the expression, `$qry->QCNUMLE' for number which is less than or equal to the expression, `$qry->QCNUMBT' for number which is between two tokens of the expression, `$qry->QCNUMOREQ' for number which is equal to at least one token in the expression, `$qry->QCFTSPH' for full-text search with the phrase of the expression, `$qry->QCFTSAND' for full-text search with all tokens in the expression, `$qry->QCFTSOR' for full-text search with at least one token in the expression, `$qry->QCFTSEX' for full-text search with the compound expression. All operations can be flagged by bitwise-or: `$qry->QCNEGATE' for negation, `$qry->QCNOIDX' for using no index.

`expr' specifies an operand exression.

The return value is always `undef'.

$qry->setorder(name, type)

Set the order of the result.

`name' specifies the name of a column. An empty string means the primary key.

`type' specifies the order type: `$qry->QOSTRASC' for string ascending, `$qry->QOSTRDESC' for string descending, `$qry->QONUMASC' for number ascending, `$qry->QONUMDESC' for number descending. If it is not defined, `$qry->QOSTRASC' is specified.

The return value is always `undef'.

$qry->setlimit(max)

Set the maximum number of records of the result.

`max' specifies the maximum number of records of the result. If it is not defined or negative, no limit is specified.

`skip' specifies the number of skipped records of the result. If it is not defined or not more than 0, no record is skipped.

The return value is always `undef'.

$qry->search()

Execute the search.

The return value is the reference to an array of the primary keys of the corresponding records. This method does never fail. It returns an empty array even if no record corresponds.

$qry->searchout()

Remove each corresponding record.

If successful, the return value is true, else, it is false.

$qry->searchget(names)

Get records corresponding to the search.

`names' specifies the reference to an array of column names to be fetched. An empty string means the primary key. If it is not defined, every column is fetched.

The return value is an array of the references to column hashes of the corresponding records. This method does never fail. It returns an empty list even if no record corresponds.

Due to the protocol restriction, this method can not handle records with binary columns including the "\0" chracter.

$qry->searchcount()

Get the count of corresponding records.

The return value is the count of corresponding records or 0 on failure.

$qry->hint()

Get the hint string.

The return value is the hint string.

$qry->metasearch(others, type)

Retrieve records with multiple query objects and get the set of the result.

`others' specifies the reference to an array of the query objects except for the self object.

`type' specifies a set operation type: `$qry->MSUNION' for the union set, `$qry->MSISECT' for the intersection set, `$qry->MSDIFF' for the difference set. If it is not defined, `$qry->MSUNION' is specified.

The return value is the reference to an array of the primary keys of the corresponding records. This method does never fail. It returns an empty array even if no record corresponds.

If the first query object has the order setting, the result array is sorted by the order.

LICENSE

 Copyright (C) 2006-2009 Mikio Hirabayashi
 All rights reserved.

Tokyo Tyrant is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License or any later version. Tokyo Tyrant is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with Tokyo Tyrant; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.