(Note clone this repository with git clone --recursive )

obigstore: multi-dimensional DB with BigTable-like data model atop LevelDB

Copyright (C) 2012-2014 Mauricio Fernandez [email protected]

obigstore is a database server + client library and associated tools. It exposes a multidimensional BigTable-like data model built on top of the Google LevelDB library, inheriting its fundamental strengths, such as fast random writes or control over the physical data layout. It can be used in a client/server setting or as an embedded database. More information can be found at obigstore.forge.ocamlcore.org.

obigstore's salient features include:

• strong data durability guarantees:
  • fully fsync'ed writes with group commit
  • data integrity ensured with CRCs at the protocol level
  • synchronous and asynchronous replication
  • online backup
• rich semi-structured data model:
  • atomic transactions (both read-committed and repeatable-read isolation levels)
  • optimistic and pessimistic concurrency control
  • asynchronous notifications
  • limited support for complex documents (BSON serialized)
  • support for composite keys (REPL and client lib)
• performance:
  • fast random writes
  • efficient range queries thanks to spatial locality
  • cross-record redundancy reduction at the page level (snappy compression)
  • fast recovery (independent of dataset size)

obigstore currently includes:

• the standalone database server
• the embeddable database library
• the client library
• a friendly REPL for interactive data manipulation
• DB dump/restore tools
• a number of benchmarking tools

Limitations

• transactions must fit in memory
• sharding on key ranges is not built-in at this point
• no automatic failover yet

Requirements

(Debian package names parenthesized)

• OCaml >= 3.12.0 (ocaml-nox)
• findlib (ocaml-findlib)
• omake to build (omake)
• GCC with C++ frontend (g++)
• OCaml Batteries (libbatteries-ocaml-dev) (>= 2.0.0)
• Cryptokit (libcryptokit-ocaml-dev)
• Lwt (liblwt-ocaml-dev)
• oUnit for the unit tests (libounit-ocaml-dev)
• extprot
• ocaml-leveldb
• atdgen (libatdgen-ocaml-dev)
• yojson (libyojson-ocaml-dev)

Building

Just

$ omake    # -j 4  for instance to parallelize the build

should do. This will build the server, client programs and libraries. You can run the tests with

$ omake test

You can then install at once the obigstore core, client, server and util libraries, as well as the executables with:

$ omake install

The generated executables are:

• obigstore: the DB server
• ob_repl: the REPL
• ob_dump: tool to dump a DB to a file
• ob_load: tool to load ob_dump's output into a DB

They are standalone (modulo the snappy and tcmalloc dependencies) and can be used simply by placing them somewhere in the PATH.