More general key-val?

[Chris00]

Since LevelDB says that ”Keys and values are arbitrary byte arrays.”, I was wondering whether general (marshallable) values are possible. The idea is to have a functor to "generate" the type specialized functions:

module Make : functor (T : sig type key type value end) -> sig
  type db
  ...
  val get : db -> T.key -> T.value option
  ...
end

(The db type is not shared between generated modules to avoid calling different get function on it.)

[mfp]

On Sun, Apr 13, 2014 at 11:57:34PM -0700, Christophe Troestler wrote:

Since LevelDB says that ”Keys and values are arbitrary byte arrays.”, I was wondering whether general (marshallable) values are possible. The idea is to have a functor to "generate" the type specialized functions:

module Make : functor (T : sig type key type value end) -> sig
  type db
  ...
  val get : db -> T.key -> T.value option
  ...
end

(The db type is not shared between generated modules to avoid calling different get function on it.)

I have implemented something similar (but safer albeit less convenient, since
it does not rely on Marshal) to this in obigstore (which is built atop
ocaml-leveldb).

type ('key, 'row) row_func =
   [ `Raw of ('key, string) key_data -> 'row option
   | `BSON of ('key, decoded_data) key_data -> 'row option ]

(* key_data holds the key and a number of "columns" (column name + value)
   associated to it *)

module type TABLE_CONFIG =
sig
  type 'a row

  module Codec : Obs_key_encoding.CODEC_OPS

  val name : string
  val row_of_key_data : (Codec.key, Codec.key row) row_func
  val row_needs_timestamps : bool
end

module Make :
  functor (M : TABLE_CONFIG) ->
  functor (OP : Obs_data_model.S) ->
sig
  val get_row : keyspace -> key -> key M.row option Lwt.t
  ...
end

This is supplemented by Obs_key_encoding, which provides simple
order-preserving serialization via combinators and type-safe range operations;
i.e., given a table User_comments whose key is a tuple of the form
(user_id * timestamp * comment_id), you can retrieve all the comments by a user with

User_comments.get_rows ks ~max_keys:10000
(User_comments.key_range_with_prefix (prefix1 user_id))

It'd be tempting to retrofit obigstore's key encoding and range operations
into ocaml-leveldb, but where to stop? At that point you might also want to
bring in support for tables (which are of course encoded using a (table_id *
key) tuple) and you end up having half of obigstore again. (Note that
obigstore can be used both in a client/server setting and as an embedded
database.)

Mauricio Fernández

[Chris00]

I meant to keep it simple! The use of marshalling is interesting if it is possible to use Marshall.to_channel so that the converted value be not entirely hold into memory. It seems that level-db offers no such possibility though...